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Abstract 


The  Cyberlink  prototype  was  developed  in  1991  as  an  alternative  controller  using  forehead 
biopotentials,  acquired  and  processed  in  a  novel  way,  as  multiple  computer  input  signals.  A 
1995  SBIR  Phase  I  feasibility  study  showed  that  operators  were  able  to  use  the  interface  for 
both  continuous  and  intermittent  computer  control  with  surprising  efficiency  and  accuracy. 
Operators  performed  successfully  within  minutes  and  accuracy  improved  with  training.  The 
measurement  site  was  easy  to  access  and  reliable  signals  were  routinely  obtained.  In  other 
studies  operators  were  able  to  demonstrate  simultaneous  independent  control  of  at  least  two 
control  signals  derived  from  the  single  forehead  site.  Discrete  control  with  under  0.2  second 
response  times  were  measured. 

The  Phase  II  effort  was  undertaken  to  transition  the  technology  to  a  mature  preproduction 
prototype.  Physiological  analyses  based  upon  literature  review  and  experiments  were 
performed  to  refine  and  expand  the  cursor  control  functionality  .  A  dry  electrode/headband 
system  was  developed.  A  low  noise  bioamplifier/microcontroller  signal  acquisition  system 
was  developed.  A  software  suite  of  training,  formatting  and  adjustment  utilities  was  created, 
fully  exploiting  WINDOWS  software  tools  to  improve  the  graphical  user  interface  (GUI).  An 
automatic  interface  calibration  scheme,  sensitive  to  individual  operator  characteristics,  was 
implemented.  WINDOWS  3.1  and  WINDOWS  95  mouse  drivers  were  developed  for 
Cyberlink  interface  control  of  third-party  software  applications. 

Through  an  extensive  program  of  research-based  design,  testing,  analysis,  redesign,  and  re¬ 
testing,  the  two-axis  cursor  control  paradigm  of  the  Cyberlink  WINDOWS  95  mouse  was 
made  more  accurate,  stable  and  robust.  At  the  same  time,  GUI  improvements  allowed 
quicker  learning  and  ease  of  use  of  the  system. 

In  a  two-axis  target  acquisition  task  involving  cursor  positioning  and  clicking  over  randomly 
appearing  targets,  subjects’  Cyberlink  target  acquisition  times  were  consistent  with  Fitts  Law 
and  on  average  about  four  times  longer  than  those  observed  with  manual  mouse  devices. 

This  is  an  encouraging  result,  particularly  when  considering  the  subjects’  limited  training  with 
the  Cyberlink  interface  and  far  greater  experience  with  the  manual  mouse. 

The  results  of  the  SBIR  Phase  II  effort  indicate  that  the  Cyberlink  interface  is  a  viable  mouse 
replacement  computer  input  device  with  far-ranging  applications.  It  can  be  used  in  the 
military  to  unload  the  pilot’s  hands  during  cockpit  management  tasks,  to  enhance  robotics 
controllers,  to  augment  human/computer  interfaces,  to  enhance  voice  actuated  hands-free 
controllers  for  such  things  as  mobile  computing,  and  as  an  assistance  and  therapeutic  training 
device  for  veterans  with  disabilities.  In  the  commercial  sector  the  interface  can  be  used  as  a 
hands-free  mouse  replacement,  for  relaxation/recreation  or  education  as  an  alternative  video 
game  or  synthesized  music  and  art  controller,  and  for  the  hands-free  operation  of  disabled  or 
special  needs  assistance  devices,  including  internet  access,  environmental  controls,  wheelchair 
control  and  muscle  stimulation  control. 
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Important  Note  to  the  Reader 


This  report  is  intended  for  general  distribution.  The  device  described  in  this  report  is  under 
continuing  development  for  commercial  applications  by  Brain  Actuated  Technologies,  Inc. 
To  avoid  unintentional  disclosure  to  parties  unknown,  those  software  and  hardware 
components  which  are  currently  proprietary  and  trade  secret  are  not  discussed  in  detail. 
Additional  information  concerning  proprietary  or  trade  secret  aspects  may  be  requested  from 
the  authors. 
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1.0  INTRODUCTION 


1.1  Background 

Contemporary  Air  Force  systems  have  reached  an  unprecedented  level  of  sophistication  and 
complexity.  Operators,  routinely  called  upon  to  manipulate  simultaneously  a  multitude  of 
buttons,  keys,  switches  and  joysticks,  are  experiencing  what  has  been  termed  “manual  control 
overload”.  Alternative,  non-manual  computer  input  devices  are  currently  being  developed  to 
lessen  cockpit  management  demands  and  provide  an  additional,  efficient  and  intuitive  operator- 
machine  interface. 

Brain  Actuated  Technologies,  Inc.  (B.A.T.)  in  Yellow  Springs,  Ohio  developed  the  first 
Cyberlink™  interface  prototype  in  1991  as  an  alternative,  brain-body  actuated,  hands-free  multi¬ 
channel  continuous  control  interface. 

By  using  biopotentials  present  on  the  surface  of  the  head  as  control  signals,  the  Cyberlink 
interface  moves  the  site  of  control  closer  to  the  brain.  The  surface  biopotentials,  defined  as  a 
"BrainBody  signal",  are  the  summation  of  electrical  activity  from  the  brain  in  the  form  of 
electroencephalalographic  (EEG)  potentials  and  from  the  body  in  the  form  of  electromyographic 
(EMG)  and  electro-oculargraphic  (EOG)  potentials. 

In  the  initial  hardware  design,  three  hydro-gel  surface  electrodes  were  affixed  to  the  operator’s 
forehead  by  means  of  a  cloth  headband.  The  center  electrode  served  as  a  reference/ground.  The 
forehead  was  selected  as  the  signal  source  due  to  the  relative  ease  and  minimal  preparation 
involved  in  obtaining  the  signal.  The  relatively  high  level  of  EMG  and  EOG  in  the  forehead 
signal  were  applied  as  additional  physiological  inputs,  facilitating  rapid  training  and  rudimentary 
control  of  the  interface. 

The  headband  signal  was  conveyed  along  a  coaxial  cable  to  a  hand-wired  prototype 
bioamplifier/analog-to-digital  converter.  The  digital  signal  was  transmitted  through  the  parallel 
(printer)  port  to  the  IBM-compatible  PC’s  CPU  where  all  signal  analysis  and  display  programs 
were  resident. 

The  core  program  employed  a  new  decoding  algorithm  (U.S.  patent  5,474,082)  developed  by 
B.A.T.  to  allow  a  more  rapid  translation  of  the  composite  BrainBody  signal  into  component 
frequency  bands  than  was  previously  possible  with  the  Fast  Fourier  Transform  (FFT).  In  the 
initial  design,  two  of  these  derived  component  frequencies,  one  a  high  theta  rhythm  at  6.25  Hz 
and  the  other  a  low  beta  at  13.25  Hz,  became  the  analog  continuous  control  signals  for  the 
vertical  and  horizontal  movement,  respectively,  of  a  two-axis  cursor  in  a  computer  maze  game 
prototype. 
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1.2  First  Studies 


In  Cyberlink  pilot  studies  conducted  in  1992,  naive  operators  were  first  given  a  biofeedback 
display  of  the  BrainBody  frequency  bands  used  for  control  resembling  a  ‘graphic  equalizer’ 
spectral  analysis.  The  height  of  the  moving  columns  in  a  bar  graph  depicted,  in  real-time,  the 
changing  derived  amplitudes  of  specific  frequency  bands.  These  ‘brain-actuated’  columns  came 
to  be  known  as  “Brainfingers.”  The  Brainfingers™  were  color-coded  to  suggest  specific 
cognitive  states  found,  in  the  biofeedback  literature  and  preliminary  subjective  Cyberlink  studies, 
to  be  associated  with  the  different  primary  EEG  bands. 


Brainfmger  Color _ Rhythm/EEG _ Associated  Cognitive  State 


blue 

low  frequency/theta  waves 

inattentive,  first  stages  of  sleep 

green 

medium  frequency/alpha  waves 

unfocused  attention,  meditative,  synthetic 

red 

high  frequency/beta  waves 

focused  attention,  concentrating,  analytical 

The  pilot  subjects  were  next  trained  in  a  one-axis  target  acquisition  task.  The  horizontal 
movement  of  a  computer-generated  pong  paddle  was  proportionally  mapped  to  the  amplitude  of 
a  13.25  Hz  Brainfmger.  The  subjects  learned  to  position  the  paddle  to  acquire  moving  targets 
(represented  as  pong  balls)  by  increasing  their  Brainfmger  amplitude  to  drive  the  paddle  to  the 
right  and  decreasing  it  to  bring  it  back  to  the  left.  The  Brainfmger  frequency  of  13.25  Hz  was 
chosen  to  correspond  to  the  frequency  of  control  used  in  the  Roll  Axis  Tracking  Simulator 
evoked  response  experiments  conducted  at  the  Armstrong  Laboratory,  Wright-Patterson  AFB, 
Ohio  (McMillan,  Calhoun,  et  al.,  1994). 

After  approximately  ten  minutes  experience  with  the  one-axis  pong  task,  the  subjects  were 
transitioned  to  a  two-axis  maze  consisting  of  50  walls.  The  horizontal  motion  of  the  game  cursor 
was  controlled  by  the  13.25  Hz  Brainfmger  while  a  6.25  Hz  Brainfinger  simultaneously 
controlled  the  vertical  motion.  After  a  couple  of  attempts,  most  subjects  succeeded  in  navigating 
through  the  maze  in  under  two  minutes  but  found  it  difficult  to  describe  the  specific  mental  and 
physical  methods  they  were  using  to  accomplish  this. 

A  series  of  studies  were  designed  to  focus  on  single-axis  control  tasks  to  illuminate  the  ‘inner 
workings’  of  Cyberlink  control.  The  most  important  of  these  were  performed  in  partnership  with 
the  US  Air  Force  and  are  reviewed  in  detail  in  the  following  sections. 
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1.3  Target  Acquisition  Study 

Under  Logicon  contract  #F4 1624-94-600,  a  study  was  run  for  the  Armstrong  Laboratory  at 
Wright-Patterson  AFB,  Dayton,  Ohio,  to  evaluate  the  effects  of  training  and  individual 
differences  on  performance  in  a  one-axis  continuous  control  task  with  the  Cyberlink  interface 
(Junker,  Berg,  Terenzio,  et.  al.,  1995).  Methods  for  evaluating  the  relative  contributions  of  EEG 
and  EMG  (as  found  in  the  forehead  biopotentials)  to  control  the  Cyberlink  interface  at  13.25  Hz 
were  developed.  Results  from  this  study  provided  insights  into  the  effects  of  extensive  training 
on  performance  with  the  Cyberlink  Interface.  Times  to  reach  asymptotic  levels  of  trained 
performance  were  obtained.  Subject  control  strategies  were  identified. 

Ten  test  subjects,  seven  male  and  three  female,  performed  the  Cyberlink  target  acquisition  ‘pong’ 
task  for  a  training  period  of  20  sessions  at  150  trials  per  session.  As  before,  the  13.25  Hz 
Brainftnger  was  proportionately  mapped  to  the  horizontal  motion  of  the  paddle.  To  help 
characterize  subject  control  methods  and  strategies,  EEG  and  EMG  data  was  collected  via  the 
Cyberlink  interface  during  task  performance.  Two  piezoelectric  vibration  transducers  mounted  in 
the  headband  just  above  the  subjects’  right  and  left  temples  provided  additional  channels  of 
signal  data  reflecting  facial  and  eye  muscle  activity. 

Four  subjects  had  trial  blocks  in  which  they  acquired  15  out  of  15  targets  and  seven  performed 
the  task  above  an  80%  accuracy  level  by  their  sixth  session.  The  majority  exhibited  a  significant 
degree  of  task  learning  over  the  course  of  the  20  sessions.  Regression  analysis  on  the  difference 
between  the  subjects'  task  accuracy  in  the  first  and  last  quarter  of  their  training  and  the  results  of 
pre-training  questionnaires  designed  to  quantify  the  subject's  attitudes/anxiety  towards  computers 
and  their  fear  of  success  indicated  significant  associations  between  these  psychological  factors 
and  task  learning. 

Subjects  were  given  both  the  ability  to  adjust  the  gain  and  response  speed  of  the  Cyberlink 
Interface,  and  the  range  sensitivity  and  centering  of  Brainfinger  magnitude  to  control  paddle 
position.  They  were  instructed  to  find  their  own  interface  adjustment  scheme  for  performing  the 
target  acquisition  task.  A  comparison  of  the  subjects'  one  axis  cursor  control  methods,  task 
accuracy  scores,  and  user  adjustment  settings  suggested  that  control  style  (EEG-based  versus 
EMG-based)  was  more  associated  with  certain  adjustment  combinations  than  with  any  particular 
level  of  task  performance. 

For  targets  to  the  right  of  center  of  the  paddle  range  of  movement,  successful  acquisition  required 
relatively  large  control  inputs  on  the  part  of  the  subjects.  When  the  Cyberlink  interface  was  set 
up  with  the  nominal  adjustment  values  many  subjects  had  to  resort  to  increased  body  activity  to 
generate  these  large  control  inputs.  This  hypothesis  was  supported  by  an  observed  increase  in  the 
derived  EMG  signal  and  the  vibration  transducer  signal. 

With  high  gain,  on  the  other  hand,  a  smaller  control  input  was  required  to  move  the  paddle  right 
of  center.  This  was  especially  observed  in  one  subject  who  chose  to  adjust  the  interface  for  high 
gain  and  slow  response  speed.  In  a  post-experiment  debriefing  questionnaire  this  subject 
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reported  that  he  was  using  primarily  mental  techniques  to  move  the  paddle,  a  contention 
corroborated  by  his  extremely  low  EMG  and  piezo-vibration  transducer  signal  levels. 

Subjects  selecting  the  nominal  settings,  on  the  other  hand,  reported  using  various  combinations 
of  physical  methods  (eye  and  facial  muscle  movements)  and  non-physical  methods 
(visualizations,  conceptual-cognitive  tasks,  emotional  state  changes,  etc.)  to  control  the  paddle. 
These  results  led  to  a  crucial  redesign  of  the  user  interface  adjustments  facilitating  and 
encouraging  user  exploration  of  the  full  range  of  control  possibilities. 


1.4  SBIR  Phase  I  Studies 

With  the  above  results  as  a  foundation,  a  Phase  1  SBIR  effort  was  undertaken  to  determine 
appropriate  directions  for  further  development  and  possible  commercialization  of  the  Cyberlink 
interface  (Junker,  Berg,  Schneider,  &  McMillan,  1995). 

A  study  was  conducted  to  evaluate  in  greater  detail  the  relative  contributions  of  various  muscle 
(EMG-based)  and  non-muscle  (EEG-based)  control  techniques  and  user-adjustment  settings  to 
Cyberlink  task  performance.  Six  subjects,  four  male  and  two  female,  performed  300  trials  over 
10  sessions  in  a  station  keeping  task  and  a  modified  pong  target  acquisition  task. 

The  response  and  control  utility  of  four  different  Brainfinger  frequencies  was  also  investigated. 
Four  representative  Brainfingers  were  chosen  to  span  the  accepted  EEG  frequency  bands:  6.25 
Hz  in  the  theta  band,  9.50  Hz  in  the  alpha,  13.25  Hz  in  the  low  beta,  and  20.25  Hz  in  the  mid¬ 
beta. 

The  pong  target  acquisition  task  was  redesigned  with  a  wider  range  of  possible  user  adjustment 
values  and  a  simplified,  more  intuitive  graphical  user  interface.  To  gain  insights  into  the  user’s 
ability  to  produce  a  constant  output  in  Cyberlink  cursor  positioning,  a  station  keeping  task  was 
created.  Brainfinger  magnitude  proportionately  controlled  the  horizontal  movement  of  a  small 
box  in  the  display.  Targets  appeared  in  the  various  locations  on  the  cursor’s  axis  of  movement 
and  the  subject  moved  the  box  over  the  target,  attempting  to  hold  a  fixed  position  until  the  next 
target  appeared. 

A  possible  approach  to  training  subjects  to  develop  a  more  subtle,  efficient,  and  internalized 
control  methodology  was  investigated.  An  additional  display  of  EMG  signal  activity  was  added 
to  each  of  the  two  task  displays.  Subjects  were  instructed  to  keep  the  EMG  signal  activity 
display  (an  indicator  of  facial  muscle  activity)  at  a  minimal  level  while  performing  the  control 
task.  Heavy  EMG  signal  activity  tended  to  cause  the  cursor  to  become  extremely  unstable  or  to 
exit  the  playing  field,  resulting  in  lowered  task  performance.  In  this  way,  high  muscle-based 
control  methods  were  negatively  reinforced.  This  investigation  provided  an  opportunity  to 
discover  what  effect  knowledge  of  EMG  activity  had  on  the  ability  to  exercise  more  subtle 
control  with  the  system. 


4 


Subjects  were  instructed  to  explore  a  large  adjustment  setting  space  and  emphasis  was  placed  on 
achieving  what  is  defined  as  "a  subtle  control  strategy".  Settings  chosen  by  the  trained  subjects 
provided  information  as  to  what  desirable  default  adjustment  settings  might  be.  It  was  possible 
to  correlate  adjustment  settings  with  performance  success.  The  experiments  provided  an 
opportunity  to  observe  interface  setting  differences  between  the  two  tasks  and  the  four 
Brainfinger  frequencies. 

After  five  hours  of  training,  one  subject  performed  both  tasks  at  a  92%  accuracy  level.  Another 
subject  performed  at  an  80%  average  success  rate  with  little  or  no  detectable  facial  muscle  or  eye 
activity  involved  in  their  control  methodology. 

For  both  tasks,  performance,  control  strategies  and  techniques,  and  interface  adjustment  settings 
were  found  to  vary  according  to  the  frequency  band  (Brainfinger)  used  for  control.  Analysis  of 
the  relationship  between  the  subject’s  control  technique,  interface  adjustment,  and  task 
performance  indicated  that  subtle,  internalized  Cyberlink  control  is  more  associated  with  specific 
adjustment  values  than  with  any  particular  level  of  task  success. 

Correlation  matrices  between  pong  paddle  and  ball  position,  EMG  signal,  and  multiple 
Brainfinger  time  histories  recorded  during  performance  of  the  target  acquisition  task  showed  that 
a  majority  of  subjects  had  episodes  where  they  appeared  to  exercise  selective  manipulation  of  the 
single  Brainfinger  used  in  the  task.  When  subjects  employed  heavy  eyebrow  lifting  and  other 
gross  facial  muscle  techniques  to  acquire  targets,  a  broad-band  response  was  observed  in  the 
signal  data,  but  the  precise  mechanism(s),  mental  or  physical,  involved  in  generating  selective 
narrow-band  responses  was  unknown. 

Signal  data  was  collected  in  a  control  experiment  in  which  the  chief  investigator  performed  a 
series  of  facial  muscle  actions  most  prominent  among  the  subjects’  observed  physical  techniques 
employed  in  Cyberlink  target  acquisition  and  station  keeping.  These  data,  although  insufficient 
to  expose  the  source  of  the  subjects’  occasional  episodes  of  narrow-band  control  frequency 
manipulation,  revealed  relationships  between  signal  patterns  and  specific  facial  muscle 
contractions.  These  relationships  suggested  the  possibility  of  applying  one  or  more  of  the  facial 
gestures  to  create  an  EMG  ‘click’  event  (functionally  analogous  to  computer  mouse  button 
clicks)  for  discrete  commands. 

A  preliminary  rule-based  auto-adjustment  program  was  designed  and  evaluated.  A  Cyberlink 
game  involving  ten  Brainfingers  manipulated  sequentially  to  acquire  a  target  moving  slowly 
along  the  x-axis,  called  “SlowBall”,  was  modified  to  provide  baseline  performance  and  signal 
data  to  an  auto-adjustment  algorithm  for  determining  user-appropriate  system  gain  and  cursor 
offset  values.  When  applied  to  the  subject’s  performances  of  the  target  acquisition  and  station 
keeping  tasks,  these  auto-adjusted  user-interface  values  seemed  to  foster  more  subtle  and 
efficient  cursor  control  techniques  without  lowering  task  accuracy. 

The  opportunity  to  explore  different  controller  designs  was  undertaken  in  the  SBIR  Phase  I 
effort.  Operators  in  Cyberlink  control  tasks,  particularly  the  station  keeping  task,  required  some 
degree  of  output  signal  averaging  to  reduce  the  negative  effect  of  cursor  instability  or  "jitter"  in 
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the  resulting  control  action.  An  enigmatic  noise  component  of  the  BrainBody  signal  was 
identified  as  the  major  contributor  to  jitter.  However,  the  signal  averaging,  or  “smoothing” 
required  to  raise  effective  signal-to-noise  ratio  to  an  acceptable  level  had  the  side  effect  of  adding 
an  unacceptable  delay  in  the  responsiveness  of  a  system. 

Various  extended  time  base  signal  averaging  methods  and  washout  and  lead  compensation  filters 
to  reduce  the  effects  of  noise  and  delays  on  system  performance  were  implemented.  None  of 
these  refinements  appeared  to  improve  operator  performance.  Fortunately,  in  the  process  of 
developing  them,  it  was  discovered  that  a  significant  portion  of  the  system  instability  andjitter 
was  a  consequence  of  unnecessarily  high  Cyberlink  interface  hardware  ambient  noise,  a  situation 
that  could  be  corrected  at  its  source.  A  new  signal  detection/amplification  system  with  a  lower 
noise  floor  and  increased  signal  sensitivity  was  projected  for  development  under  the  SBIR II 
effort  reported  here. 


1.5  SIRE  Studies 

In  the  fall  of  1995,  a  joint  study  was  conducted  in  the  Synthesized  Immersion  Research 
Environment  (SIRE)  at  the  USAF  Armstrong  Laboratory,  Wright-Patterson  AFB,  Dayton,  Ohio. 
Teams  from  Logicon  Technical  Services,  Inc.,  Armstrong  Laboratory,  and  B.A.T.,  Inc. 
collaborated  to  assess  the  Cyberlink  interface  as  an  alternative  controller  for  virtual  environments 
and  tactical  airborne  applications  (Nelson  et  al.,  1996a  and  Nelson  et  al.,  1996b).  Cyberlink 
performance  and  skills  acquisition  were  evaluated  in  a  discrete  reaction  time  task  and  a  one-axis 
continuous  tracking  task  in  an  immersive  virtual  reality  flight  simulator. 

In  work  leading  up  to  the  SIRE  studies  an  EMG  pulsatile  capability  was  demonstrated  as  a  viable 
input  signal  for  discrete  commands.  Most  subjects  quickly  learned  to  create  a  timely  EMG 
double  pulse.  In  the  prototype,  a  double  pulse  was  chosen  as  the  criteria  for  the  EMG-click  event 
to  reduce  the  likelihood  of  unintentional  ‘clicking.’ 

In  subsequent  pilot  tests  two  important  discoveries  were  made: 

1)  when  subjects  were  presented  with  a  real-time  history  trace  of  their  EMG  signal  with 
the  click-event  threshold  represented  as  a  dotted  horizontal  line  above  the  trace  and  a  computer¬ 
generated  example  of  the  requisite  click  waveform  (appearing  as  a  spike  in  the  EMG  trace)  they 
could  learn  almost  immediately  to  execute  EMG-clicks  more  rapidly  and  just  as  reliably  with  a 
single  EMG  pulse,  and 

2)  a  variety  of  facial  muscle  actions  could  be  applied  to  produce  the  requisite  click 
waveform  in  the  EMG,  ranging  from  a  short,  concise  eyebrow  lift  to  a  deliberate,  hard  blink. 

This  single  pulse  EMG-click  was  adapted  to  provide  the  control  input  in  the  SIRE  discrete 
control  study.  Five  naive  subjects,  three  males  and  two  females,  performed  10  experimental 
sessions  at  50  trials  per  session  of  a  simple  reaction  task;  generating  Cyberlink  EMG-clicks  as 
quickly  as  possible  in  response  to  a  critical  visual  stimulus  appearing  among  a  suite  of  avionics 
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displays  on  a  computer  monitor.  The  subjects  were  given  general  instructions  in  ‘EMG-clicking’ 
and  were  encouraged  to  explore  various  click  techniques  to  find  their  own  preferred  method,  but 
received  no  training  in  any  specific  technique. 

The  results  confirmed  that  discrete  Cyberlink  control  is  intuitive  and  can  be  mastered  quickly. 
With  no  special  coaching,  subjects’  mean  reaction  times  improved  dramatically  over  their  initial 
experimental  sessions,  reaching  asymptotic  performance  before  the  fifth  experimental  session. 
Response  accuracy  was  found  to  be  extremely  high,  approximately  98%,  and  reaction  times  fell 
between  180-200  msec,  a  range  considered  to  be  the  limit  of  simple  reaction  time  to  visual 
stimuli  (Keele,  1986).  Several  subjects  actually  achieved  15-20%  faster  reaction  times  with  the 
Cyberlink  EMG-click  than  with  a  manual  button.  The  width  or  duration  of  the  subjects’  EMG- 
click  responses  decreased  approximately  47  msec  over  the  experimental  sessions,  indicating  an 
improvement  with  training  in  the  precision  and  efficiency  of  their  EMG-click  technique. 

Of  particular  interest  in  this  study  were  the  results  of  a  self-report  inventory,  the  Yoshitake 
Symptoms  of  Fatigue  Scale  (Yoshitake,  1978),  indicating  that  the  Cyberlink  interface  can  be  used 
for  tasks  requiring  the  production  of  a  high  density  of  discrete  commands  and  responses  without 
fatiguing  the  user. 

In  the  continuous  tracking  experiment  subjects  used  the  13.25  Hz  Brainfinger  to  navigate  through 
a  series  of  hoops  defining  a  flight  path  projected  on  a  40-foot  virtual  reality  dome.  This  was  a 
continuous  control  task  on  the  x-axis  functionally  similar  to  the  pong  task,  but  with  the  added 
visual  feedback  and  realism  of  a  large  scale,  fully  immersive,  first-person-perspective  graphic 
display  and  simulated  roll  motion  in  the  direction  of  travel  (right  or  left).  To  achieve  more 
stability,  a  null-zone  was  added  to  the  basic  lateral  control  paradigm  whereby  Brainfinger 
magnitudes  exceeding  an  upper  power  threshold  initiated  rightward  movement,  those  falling 
below  a  lower  power  threshold  initiated  leftward  motion,  and  those  hovering  in  the  space 
between  the  two  thresholds  initiated  no  lateral  movement. 

12  naive  subjects,  ten  males  and  two  females,  receiving  only  generalized  instructions  in 
Cyberlink  continuous  control  and  no  coaching  in  specific  techniques,  performed  100  trials  over 
ten  sessions,  completing  a  NASA  Task  Load  Index  (NASA-TLX;  Hart  &  Staveland,  1988)  at  the 
end  of  each  session. 

Tracking  performance,  as  measured  by  RMS  error  and  percent  time  on  target,  improved 
dramatically  across  the  sessions.  In  some  cases,  asymptotic  performance  still  hadn’t  been 
achieved  by  the  subject  at  the  end  of  the  study  although  percent  of  hoops  traversed  had  exceeded 
90%.  In  all  cases,  self-reported  workload  ratings  decreased  significantly  across  the  sessions. 

The  results  of  a  Cyberlink  skill  retention  follow-up  study  were  also  promising.  In  all  measures  of 
task  accuracy  and  efficiency,  the  subjects  resumed  at  their  prior  level  of  performance  after  a  six 
month  hiatus,  suggesting  a  high  retention  rate  for  acquired  Cyberlink  control  skills.  Similar  good 
retention  of  manual  tracking  skills  has  been  reported  in  a  variety  of  studies  (Poulton,  1974). 
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1.6  Prelude  to  SBIR  Phase  II 


SBIR  Phase  I  follow-up  studies  with  other  Cyberlink  tasks,  conducted  in  late  1995,  continued  to 
demonstrate  the  general  utility  of  the  four  Brainfmgers  already  applied  to  continuous  control  and 
revealed  several  more  Brainfinger  candidates.  Phase  I  SBIR  results  had  suggested  that  certain 
control  techniques  were  Brainfinger-specific.  Follow-up  studies  revealed  that  Brainfmgers  at 
the  low  frequency  end  of  the  BrainBody  spectrum  (1.1-  6.25  Hz)  seemed  to  be  particularly 
responsive  to  ocular  movements  whereas  the  high  frequency  Brainfmgers  (20.25  -  25.5  Hz) 
appeared  to  respond  more  to  subtle  scalp  and  eyebrow  movements,  as  well  as  what  could  be 
called  a  purposeful,  yet  non-muscular,  focus  of  attention. 

By  the  end  of  1995,  nearly  a  hundred  operators  had  used  the  Cyberlink  system  in  a  variety  of 
control  tasks  and  studies.  This  research  established  that  the  forehead  signals  used  for  continuous 
and  discrete  control  were  consistent  and  intelligible  and  that  they  could  be  affected  by  most  users 
with  some  degree  of  intention  and  frequency-selectivity  after  minimal  training.  In  addition, 
Cyberlink  control  did  not  produce  apparent  undue  fatigue  or  negative  side  effects,  even  after 
prolonged  use. 

The  success  at  different  frequencies,  the  observation  that  different  control  techniques  are 
effective  at  different  frequencies,  and  enhancements  in  the  EMG-click  algorithm  arising  from  the 
SIRE  study,  suggested  the  possibility  of  a  2-axis  continuous  plus  discrete  alternative  controller 
application  of  the  BrainBody  actuated  technology. 

As  a  result,  the  primary  objectives  of  this  SBIR  Phase  II  effort  were  formulated:  to  develop  the 
controller  interface  described  above  and  to  transition  it  from  a  proof-of-concept  prototype  to  a 
practical,  fully  functional,  hands-free,  pre-production  prototype  input  device. 

Achieving  these  objectives  depended  on  further  development  of  the  BrainBody  actuated 
technology  foundations.  To  expand  the  functionality  and  improve  the  accuracy  of  the  controller 
and  to  develop  suitable  applications  to  operational  settings,  a  more  detailed  understanding  of  the 
forehead  control  signals  and  their  muscle,  brain,  and  ocular  sources  was  required. 
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2.0  TOWARDS  A  BETTER  MODEL 


2.1  SBIR  Phase  II  Literature  Review 

A  comprehensive  library  and  internet  search  of  the  pertinent  EEG,  EMG,  and  EOG  biofeedback 
and  source  localization  literature  was  undertaken  in  four  phases  from  autumn  1995  to  winter 
1996  to  gain  an  in-depth  understanding  of  the  nature  and  physiological  origins  of  the  forehead 
biopotentials.  The  topics  researched  included: 

1)  Multi-channel  EEG  brain  topographical  mapping  and  other  source  localization  studies 
carried  out  in  the  process  of  isolating  EEG  origins  of  surface  electrode  signals  (Marks  &  Issac, 
1995;  Poline  &  Mazoyer,  1994;  Wilson,  1994;  Lancaster,  Fox,  et  al.,  1994;  Inouye  et  ah;  1993, 
Baumgartner,  1993;  Luders  et  ah,  1992;  Montgomery  et  ah,  1992;  Gevins  &  Illes,  1991;  Wong, 
1991;  Weiss  et  ah,  1987;  Hjorth,  1986;  Fox  et  ah,  1985;  Uemura  et  ah,  1983;  Gevins  et  ah,  1980 
and  Lindsley,  1952), 

2)  Comparison  studies  of  source  localization  from  the  magnetoencephalogram  (MEG)  in 
simultaneous  EEG  studies  (Baumgartner,  1993;  Wikso  et  ah,  1992;  Kaufman,  1993  and  Deecke 
et  ah,  1982), 

3)  Cognitive  state  and  physiological  correlates  of  specific  EEG  rhythms  and  brain 
potentials  (Linden  et  ah,  1996;  Steriade,  1996;  Munk  et  ah,  1996;  Ganis  et  ah,  1996;  Miller, 

1996;  Gevins  et  ah,  1995;  Pfurtscheller  &  Pregenzer,  1995,  Poijesz  &  Begleiter,  1995;  Williams, 
Rippon  et  ah,  1995;  Williams,  Zaveri,  et  ah  1995,  Birbaumer  et  ah,  1994;  Freeman  &  Barrie, 
1994;  Jobert,  1994;  Jokeit  &  Makeig,  1994;  Pope  et  ah,  1994;  Sterman  et  ah,  1994;  Sterman  et 
ah,  1993;  Inouye  et  ah,  1993;  Freeman,  1993;  Freeman  &  Barrie,  1993;  Dujardin  et  ah,  1993; 
Besson  &  Kutas,  1993;  Sobotka  et  ah,  1992;  De  Toffol  et  ah,  1992;  Paller  &  Kutas,  1992; 
Sterman  et  ah,  1992;  Bullock,  1992;  Gevins  et  ah,  1991;  Klorman,  1991;  Linden  et  ah,  1991; 
Sumathy  &  Krishnan,  1991;  Mast  &  Victor,  1991;  Henriques  et  ah,  1990;  Kutas  &  Van  Petten, 
1990;  Lubar,  1989;  Pfurtscheller  &  Berghold,  1989;  Kutas  &  Hillyard,  1988;  Schwartz,  1987; 
Speil,  G.,  1987;  Banquet,  1984;  Kasamatsu  &  Hiri,  1984;  Gratton,  Coles,  &  Donchin,  1984; 
Elbert  et  ah,  1984;  Gevins  &  Schaffer,  1980;  Kutas  &  Hillyard,  1980;  Bird  et  ah,  1978;  Cazard, 
1976  and  Lindsley,  1952), 

41  Biofeedback  literature  providing  additional  data  on  the  frequency  ranges  of  various 
bioelectric  signals  (Williams,  Rippon  et  ah,  1995;  Blanco  et  ah,  1995;  Othmer  et  ah,  1991  and 
Grossman  &  Weiner,  1966), 

5)  Techniques  for  isolating  EMG.  EEG.  and  EOG  signals  and  automated  recognition  of 
artifact  and  system  noise  in  EEG.  EOG,  and  EMG  (Blanco  et  ah,  1995;  Verleger,  1993;  Zheng  et 
ah,  1993;  Takano  et  ah,  1993;  Roessgen  et  ah,  1993;  Gasser  et  ah  1986;  Berg,  1986;  Barlow, 
1983;  De  Oliveira,  1983;  Gratton,  Coles,  &  Donchin,  1983;  Bird  et  ah,  1978;  Gevins  et  ah,  1977 
and  Carpenter,  1948), 
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6)  EEG  slow  potentials  (SP)  and  response-oriented  cognition:  contingency  evaluation, 
attention,  expectancy,  and  response  preparation  (Cooper  et  al.,  1989;  Achermann  &  Borbely, 
1987;  Deecke  et  al.,  1982;  Deecke  &  Grozinger,  1976;  and  Gotman  et  al.,  1973), 

7)  Neural  net-aided  recognition  of  sensory  and  cognitive  processing  in  the  EEG  and 
automated  on-line  EEG  pattern  recognition  (Anderson  et  al.,  1995a-c;  Pfiirtscheller  &  Pregenzer, 
1995;  Flotzinger,  Pfiirtscheller,  et  al.,  1993;  Peltoranta  &  Pfiirtscheller,  1994;  Dorffiier,  1995; 
Gevins  et  al.,  1995;  Veselis,  1993;  Jando  et  al.,  1992;  Bartlett,  Makeig,  et  al.,  1995;  Gabor,  1992; 
Gevins  &  Leong,  1992;  Gevins  &  Illes,  1991;  and  Nakamura  et  al.,  1992), 

8)  EEG  and  EOG  monitoring  of  vigilance,  alertness  and  sustained  attention  and  neural 
human-system  interface  technology  (Makeig  &  Jung,  1995;  Jung  &  Makeig,  1995a-b;  Jung  & 
Makeig,  1994;  Makeig  &  Inlow,  1992;  Gevins  et  al.,  1991;  Makeig  et  al.,  1990;  Moore-Ede  et 
al.,  1989;  and  Matousek  &  Petersen,  1983), 

9)  EMG/EOG  biofeedback  and  frontalis  muscle  tension/relaxation  (Heetderks  & 
Schwartz,  1995;  Harwin  et  al.,  1995;  and  Grossman  &  Weiner,  1966), 

10)  EMG-based  discrete  control  and  reaction  time  (Nakajima,  1995;  Ulrich  &  Miller, 
1994;  Bartz,  1979  &  Laszio  et  al.,  1977), 

1 1)  Visual  imagery,  motor  processes  and  EEG  activity  (Williams,  Rippon  et  al.,  1995; 
Marks  &  Issac,  1995;  Issac  &  Marks,  1994;  Weiss  et  al.,  1987;  Farah,  1988  &  ’84  and  Farah  et 
al.,  1985;  Marks  et  al.,  1985,  Davidson  &  Schwartz,  1977;  and  Short,  1953), 

12)  Perceptual-cognition,  visual  imagery,  visuo-spatial  tasks,  intense  vigilance,  voluntary 
planned  (willful)  decisions  correlating  with  enhanced  beta  2  and  gamma  band  activity  and 
suppressed  alpha  in  the  prefrontal  cortex  (Hotz,  1996;  Chorlian  et  al.,  1995;  Searight  et  al.,1995; 
Passingham,  1995;  Goldman,  1995;  Williams,  Rippon  et  al.,  1995;  Ford  et  al.,  1994;  Birbaumer 
et  al.,  1994;  Freeman,  1993;  Freeman  &  Barrie,  1993;  Oscar-Berman  &  Hutner,  1993;  Jagust  et 
al.,  1992;  Paller  &  Kutas,  1992;  Shimamura  et  al.,  1988;  Haxby  et  al.,  1986  and  Shepard  & 
Podgomy,  1978), 

13)  The  use  of  Raven  Progressive  Matrices  (selection  of  an  abstract  shape  from  a  set  of 
four  alternatives)  as  a  cognitive  perceptual  task  and  visuo-spatial  stimulus  to  evoke  beta  2  and 
gamma  band  EEG  responses  in  the  prefrontal  cortex  (Rippon,  1990  and  Scheblanova,  1984). 

14)  EEG-based  control  interfaces  (McMillan,  1995,  McMillan  et  al.,  1995;  Wolpaw  et  al., 
1991;  Junker  et  al.  1988  and  Junker  et  al.,  1987),  and 

15)  Existing,  contact  dry  electrode  technology  (Taheri,  1995;  Prutchi  et  al.,  1993; 
Bourland  et  al.,  1978;  Betts  &  Brown,  1976;  David  &  Portnoy,  1972;  Bergey  et  al.,  1971;  Potter 
&  Menke,  1970,  and  Johnson  &  Allred,  1968). 
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2.2  Literature  Search  Results  and  Theoretical  Model  Formulation 

Although  much  of  the  literature  disagreed  on  certain  topics  (e.g.,  the  precise  frequency  ranges  of 
the  primary  bands  of  the  EEG  spectrum,  the  degree  to  which  various  source  localization  methods 
can  be  said  to  truly  isolate  desired  electrocortical  signals,  etc.),  there  was  a  meaningful  consensus 
on  several  major  points  crucial  to  this  SBIR II  effort: 

la)  In  the  EEG,  activity  in  the  delta  through  theta  bands  (approximately  .5  Hz  to  7.5  Hz)  tends  to 
increase  as  focus  of  attention  or  level  of  arousal  diminishes,  i.e.,  during  extreme  drowsiness,  the 
early  or  “slow  wave”  stages  of  sleep,  when  there  is  structural  damage  to  areas  of  the  brain 
associated  with  the  purposeful  focus  of  attention,  or  when  the  subject  has  Attention  Deficit/ 
Hyperactivity  Disorder  (AD/HD). 

lb)  Blinking  and  eye  movements  produce  EOG  transients  in  the  delta  and  low  theta  bands  which 
are  difficult  to  distinguish  and  separate  from  low  frequency  EEG. 

2)  EEG  slow  potentials  (possibly  reflected  in  time-averaged  magnitudes  of  very  low  frequency 
Brainfmgers)  are  highly  correlated  with  the  velocity  of  target  movement  in  a  visual  pursuit 
tracking  task. 

3a)  Alpha  rhythms  (approximately  8  to  13  Hz)  tend  to  become  enhanced  in  the  prefrontal  cortex 
(and  other  brain  locations)  when  subjects  are  aware  but  relaxed  and  when  their  attention  is 
diffused  or  defocused,  their  eyes  are  defocused,  or  when  they  are  immersed  in  a  meditative  or 
hypnogogic  state. 

3b)  Alpha  rhythms  are  suppressed  in  the  prefrontal  cortex  (and  other  locations)  during  the 
mental  rotation  of  imaginary  objects,  concentration  on  mental  arithmetic,  tasks  with  high 
cognitive  demands,  situations  of  high  and  complex  task  load,  the  self-generation  of  imagery, 
complex  information  processing,  and  during  periods  of  effortful  or  concentrating  attention  and 
intense  vigilance  or  arousal. 

4a)  Beta  2  rhythms  (approximately  20  to  30  Hz)  and  Gamma  rhythms  (greater  than  30  Hz)  are 
enhanced  in  the  prefrontal  cortex  (and  other  locations)  during  activities  or  stimuli  that  tend  to 
suppress  alpha:  the  effortful  focus  of  attention  or  intense  vigilance,  the  planning  of  the  order  and 
timing  of  future  behavior  based  on  recent  past  experience,  mental  rotation  and  other  visuo-spatial 
and  cognitive  perceptual  tasks  such  as  Raven  Progressive  Matrices,  complex  and  periodic  music 
stimuli,  REM  or  “active  information-processing”  sleep,  etc. 

4b)  Overall  power  and/or  irregularity  (entropy),  in  the  beta  2  and  gamma  bands  also  increases 
during  calculation  processing,  complex  visual-memory  problem  solving,  visual  feature  binding 
and  segregation,  adaptive  behavior  based  on  ‘shifting  rule  sets’,  abstracting  (i.e.,  ascertaining 
principles  or  rules  governing  a  particular  task)  and  visual  imagery;  abilities  all  found  to  be  greatly 
impaired  in  people  suffering  from  structural  damage  to  the  prefrontal  cortex  and  the  frontal 
system  due  to  lesions,  prolonged  alcoholism  or  other  causes. 
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4c)  Power  in  the  gamma  and  beta2  bands  is  attenuated  during  deep  dreamless  sleep  and  central 
anesthesia  and  when  the  subject  has  AD/HD  or  certain  forms  of  Autism. 

This  information  was  evaluated  to  generate  the  following  hypotheses  specific  to  the  various 
control  signals  obtainable  (or  believed  to  be  obtainable)  by  the  Cyberlink  Interface: 

1)  HYPOTHESIS  THETAi  -  long-term,  time-averaged  magnitude  increases  of  the  low 
frequency  Brainfingers  will  reflect  certain  internal  subjective  states  known  to  enhance 
EEG  power  in  the  theta  or  delta  bands,  however... realistically,  operators  can  not  be  expected 
to  learn  to  position  the  2-axis  cursor  (or  other  features  controlled  by  a  low  frequency  Brainfmger) 
by  shutting  down  their  higher  brain  functions,  becoming  extremely  drowsy,  or  entering  slow- 
wave  sleep. 

2)  HYPOTHESIS  THETA2  -  short-  and  mid-term  time-averaged  magnitudes  of  low 
frequency  Brainfingers  (corresponding  to  the  theta  and  delta  EEG  bands)  can  be  increased 
in  an  intentional  and  timely  manner  through  ocular  activity,  such  as  lateral  eye  movements 
and  high-density  blinking,  for  reliable  and  selective  control  of  one  axis  of  the  2-axis  cursor. 

3)  HYPOTHESIS  DELTA-  time-averaged  magnitudes  of  very  low  frequency  Brainfingers, 
which  may  reflect  EEG  slow  potentials,  may  unintentionally  correlate  with  displayed 
cursor  velocity.  As  subjects  unconsciously  visually  track  the  cursor  across  the  display  they  may, 
in  effect,  be  evoking  a  low  frequency  Brainfmger  response  which  in  turn  may  create  spurious 
cursor  movement.  Therefore,  certain  very  low  Brainfmger  frequencies  may  need  to  be  avoided 
for  positioning  the  cursor. 

4)  HYPOTHESIS  ALPHA  -  mid-  and  long-term,  time-averaged  magnitude  increases  of  the 
mid-frequency  Brainfingers  (as  detected  on  the  forehead/pre-frontal  site)  will  reflect 
certain  intentional  subjective  state  changes  that  enhance  or  suppress  power  in  the  EEG 
alpha  bands.  Specifically,  meditation/relaxation  and  the  volitional  defocusing  of  visual 
attention  should  prove  to  be  practical  methods  for  controlling  features  set  to  a  mid-frequency 
Brainfmger  in  situations  that  are  not  time-sensitive  (e.g.  the  modulation  of  musical  events). 

5)  HYPOTHESIS  BETA/GAMMAi  -  the  effortful  focus  of  attention  on  a  perceptual 
cognitive  task  presented  on  the  display,  concentration  on  visual  imagery  or  mental  rotation, 
or  listening  to  complex  and  periodic  music  stimuli  can  be  expected  to  evoke  mid-  and  long¬ 
term  time-averaged  magnitude  increases  in  the  high  frequency  Brainfingers  (as  detected  at 
the  forehead/prefrontal  site)  corresponding  to  similar  known  responses  in  the  Beta2  and 
Gamma  EEG  bands.  Specifically,  the  above  responses  may  prove  to  be  practical  methods  for 
control  of  features  set  to  a  high-frequency  Brainfmger  in  situations  that  are  not  time-sensitive. 
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6)  HYPOTHESIS  BETA/GAMMA2  -  the  effortful  focus  of  attention  on  a  perceptual 
cognitive  task  presented  on  the  display,  concentration  on  visual  imagery  or  mental  rotation, 
or  listening  to  complex  and  periodic  music  stimuli  can  be  expected  to  evoke  an  increase  in 
the  entropy  or  irregularity  of  the  high  frequency  Brainfingers  (as  detected  on  the 
forehead/prefrontal  site)  corresponding  to  similar  known  responses  in  the  Beta2  and 
Gamma  EEG  bands.  This  irregularity  might  be  characterized  as  a  ‘spread’  in  the  variability  of 
the  sampled  Brainfmger  magnitude  values  over  a  sliding  sample  window  and  used  to  create 
selective,  efficient,  non-muscular  control  of  features  in  situations  that  are  not  time-sensitive. 


7)  HYPOTHESIS  BETA/EMG  -  subtle  contractions  of  the  frontalis,  masseter,  temporalis, 
and  other  facial  muscles  within  a  certain  proximity  about  the  sensor  locus,  will  cause 
immediate  transients  and  short-term  time-averaged  magnitude  increases,  most  pronounced 
in  the  high  frequency  Brainfingers  (above  18  Hz),  affording  rapid,  reliable  and  selective 
control  of  one  axis  of  the  two-axis  cursor. 


2.3  Testing  and  Refining  the  Model 

The  seven  critical  hypotheses  were  tested  through  a  series  of  pilot  studies  designed  to  create  an 
ever-expanding  database  on  the  origin  and  optimization  of  BrainBody  control  signals  and  to 
evolve  and  refine  the  theoretical  model. 

1)  Test  of  Hypothesis  Thetai:  Sleep  Study  -  an  interesting  study  confirming  Hypothesis  THETAi 
was  conducted  with  a  severe  brain  trauma  patient  at  a  Mental  Retardation/Developmental 
Disability  center  in  Akron,  Ohio.  This  man,  a  completely  “locked-in”  quadriplegic,  exhibited  a 
tendency  to  become  agitated  or  distressed  when  aroused,  but  to  fall  asleep  rapidly  and 
automatically  upon  relaxation.  He  was  connected  to  the  Cyberlink  and  a  special  brain-actuated 
music  program  was  engaged  which  rewarded  low  beta2  activity  (interpreted  as  low  relative  time- 
averaged  magnitudes  of  the  high  frequency  Brainfingers)  and  high  theta  activity  (low  frequency 
Brainfingers)  with  relaxing  and  soothing  music. 

His  experience  with  the  Cyberlink  became  a  cycle  of  immediate  relaxation  followed  by  very  high 
observed  power  in  the  theta  band  (displayed  as  low  frequency  brain  fingers)  and  sleep,  followed 
by  sudden  arousal  (his  caretaker  would  then  wake  him  up),  followed  by  relaxation  as  he  again 
responded  to  the  music  program,  followed  by  high  theta  and  sleep,  and  so  on.  This  low 
frequency  Brainfmger  control  technique  was  judged  to  be  too  impractical  for  most  operational 
settings. 

2)  Test  of  Hypothesis  Theta2:  Eve  Motion  Studies  -  in  one  experiment,  the  chief  investigator 
modified  a  Cyberlink  headband  harness  to  allow  placement  of  the  sensors  on  alternate  sites 
around  the  eye.  Signals  collected  during  basic  ocular  motion  (upward/downward  glancing, 
left/right  glancing,  blinking,  etc.)  were  compared  for  each  of  the  alternate  sites.  In  another 
experiment,  ten  subjects  performed  a  more  varied  series  of  eye  motions  with  the  sensors  on  their 
forehead  in  the  standard  headband.  The  results  of  both  experiments  suggested  that  lateral  eye 
movements  produced  more  reliable  and  prominent  transients  in  the  low  frequency  Brainfingers 
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(measured  at  the  forehead  site)  than  either  vertical  eye  movements  or  blinking.  As  a  result,  the 
design  of  the  cursor  control  model  was  modified  (see  following  section). 

3)  Tests  of  Hypothesis  Alpha: 

a)  Meditation/Relaxation  Study  -  long-term  BrainBody  and  EMG  signal  data  was  collected  from 
subjects  wearing  the  Cyberlink  interface  while  meditating.  The  subjects  included  two  professors 
of  Eastern  meditation  techniques  at  local  colleges  and  several  of  their  students.  Non-meditation 
experimental  controls  included  reading  text  silently,  closing  eyes  and  listening  to  a  recorded 
voice,  and  looking  at  pictures.  Analysis  of  the  signal  data  confirmed  that  significant  long-term 
increases  in  the  time-averaged  magnitudes  of  the  mid-frequency  Brainfmgers  (particularly  those 
corresponding  to  low  alpha)  accompany  self-reported  episodes  of  light  or  deep  meditation. 

b)  Optical  Defocusing  Experiment  -  Before  confirmation  was  found  in  the  literature  search,  the 
chief  investigator  noted  a  connection  between  his  alpha  Brainfmger  response  and  the  intentional 
defocusing  of  his  eyes.  Whenever  he  defocused  his  eyes  he  observed  a  sudden  increase  in  the 
displayed  height  (magnitude)  of  his  mid-frequency  Brainfmgers.  To  determine  how  general  this 
effect  might  be,  ten  subjects  looked  at  stereograms  (composite  ‘3D’  images  that  require  self¬ 
adjustment  of  visual  focal  length  to  see  the  imbedded  image)  while  their  BrainBody  signal  data 
was  collected  with  the  Cyberlink.  The  results  in  this  visual  defocusing  experiment  were 
ambiguous  because  only  two  subjects  reported  they  could  see  the  embedded  stereogram  images. 
In  subsequent  studies,  approximately  30%  of  the  sample  population  observed  they  were  able  to 
produce  the  ‘alpha  boost’  response  by  defocusing  their  eyes. 

4)  Test  of  Hypotheses  Beta/Gammai-2  -  Ten  subjects  performed  three  cognitive  perceptual  tasks 
(Raven  Progressive  Matrices),  silently  read  two  lines  of  written  text,  and  relaxed  with  their  eyes 
closed  and  open  (to  provide  baselines)  while  their  BrainBody  signals  were  being  recorded  with 
the  Cyberlink  interface  and  a  special  data  collection  program. 

As  the  subjects  performed  the  tasks,  a  video  camera  monitored  their  faces  and  their  signal  data 
was  written  to  a  text  file  for  subsequent  analysis  in  MS  Excel  and  visual  inspection  in  a  multi¬ 
channel  signal  playback  program.  The  video  camera  output  was  synchronized  and  superimposed 
with  the  signal  playback  display  using  a  ‘genlock’  device  providing  evidence  of  any  concurrent 
facial  muscle  activity  to  aid  in  evaluating  the  purity  of  the  collected  prefrontal  EEG  signals. 

The  results  showed  that  nine  subjects  produced  a  significant  increase  above  their  baselines  in  the 
variability  of  the  35  Hz  and  40  Hz  Brainfmger  magnitude  values  over  a  sliding  sample  window 
(of  a  particular  duration)  concurrent  with  the  presentation  of  the  Raven  Matrices. 

As  a  result  of  these  and  other  tests  designed  to  clarify  points  that  had  remained  ambiguous  after 
the  literature  review,  the  Cyberlink  control  theoretical  model  had  become  more  rigorous, 
facilitating  and  informing  improvements  in  the  design  of  what  came  to  be  known  as  the  two-axis 
‘control  paradigm.’ 
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2.4  Evolving  The  Control  Paradigm 

In  considering  the  original  (pre-SBIR  II)  two-axis  control  model  in  light  of  the  new  data,  and  as 
newly  developed  low-noise  hardware  permitted  higher  controller  gain,  the  fundamental  design 
elements  of  the  control  paradigm  (CP)  were  identified  and  expanded. 


Elements  of  Two- Axis  Control  Paradigm  Design 


CP  Element 


Definition 


#1: 

Signal  to  Axis 
Mapping 
(for  Continuous 
Control) 


Which  forehead  signal  controls: 
X  (lateral  cursor  movement)  ? 
Y  (vertical  cursor  movement) 


? 


#2: 

Motive  Signal 
Characteristic 


Which  attribute  change  in  the  signal 
interest  is  used  to  generate  cursor 
movement  ? 


Possibilities  -  * Tested  as  of  mid-SBIR  II 


of 


a)  *X  =  13.25  Hz  low  beta  Brainfinger  & 
*Y  =  6.25  Hz  high  theta  Brainfinger 

b)  X  =  any  Brainfinger  at  1  -  40  Hz,  or 

EMG,  or  EOG  & 

Y  -  any  of  the  above,  not  same  as  X 


a)  *  time-averaged  magnitudes  (powerO 

b)  *median/upper  and  lower  quartile  (power2) 

c)  ‘spread’  of  magnitude  samples  (chaos) 

d)  waveform  analysis  (shape) 


#3: 

Cursor  Motion 
Formula 


In  what  direction,  proportion,  and  velocity  will  the 
cursor  move  in  response  to  a  given  attribute  change  in 
the  signal  of  interest  (motive  signal  A)? 


#4: 

Cursor  Response 
Adjustment 
Method(s) 


Presence  and  application  of  user-defined  and/or  auto- 
adjusted  interface  parameter  settings:  how  far  (gain), 
at  what  velocity  (cursor  speed),  centered  at  which 
display  position  in  the  axis  of  motion  (offset),  will  the 
cursor  move  in  response  to  a  given  motive  signal 
attribute  change? 


#5: 

Cursor  Behavior 
During  Defined 
Signal  Stasis 
(zero  A  ) 


Function  and  proportions  of  a  ‘null  zone’  or  non-control 
region  within  which  the  motive  signal  A  is  defined  as  a 
zero  control  input  and  the  cursor  motion  to  be  produced 
during  these  episodes  of  zero  input. 


#6: 

Mapping  of 
Discrete 
Command(s)  to 
Mouse  Button 
and  Other 
Program  Inputs 


Which  signal  transients  making  which  pre-defined 
power  threshold  crossings  will  activate  which  Mouse 
button,  keyboard,  cursor  motion/response  attributes,  or 
other  program  functions? 


a)  * proportional  -  cursor  position  is  direct 

function  of  Brainfinger  magnitude. 

b)  *  mono-velocity  -  cursor  moves  at  fixed 
speed  when  upper  or  lower  magnitude 
thresholds  exceeded. 

c)  *  various  proportional/velocity  hybrids 

d)  *  velocity  -  cursor  velocity  is  a  direct 
function  of  magnitude. 


a)  *user  adjusted  cursor  offset,  speed  &  gain 

&  no  continuous  auto-adjust 

b)  *user  adjusted  offset  and  cursor  speed 

with  continuous  auto-adjusted  gain 

c)  *  user-adjusted  overrides  of  auto- 
calibrated  initial  offset  and  gain  values 


a)  * original  -  cursor  stationary  when  motive 
signal  within  user-defined  null  zone. 

b)  linear  -  linear  pre-programmed  motion 
when  motive  signal  within  null  zone. 

c)  polar  -  radial  pre-programmed  motion 
when  motive  signal  within  null  zone. 

a)  *EMG  spikes  activate  left  Mouse  button 
clicks  and/or  program  item  selections. 

b)  *EEG  spikes  at  a  user-defined  Brainfinger 
frequency  activate  left  Mouse  button 
clicks  and/or  program  item  selections. 

c)  *EOG  spikes  activate  left  Mouse  button 
clicks  and/or  program  item  selections 

d)  Additional  transients  and  waveforms 
activating  cursor  and  other  functions 
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Initial  Development  of  the  Two-Axis  Control  Paradigm 

With  the  expanded  control  model  as  a  blueprint,  the  following  design  element  revisions  and 

enhancements  were  implemented: 

Revised  Element  #1 : 

Lateral  (X)  Brainfinger  Re-mapping  -  Prior  to  the  SBIR  Phase  II  effort,  the  prototype 
two-axis  control  model  used  a  low  beta  Brainfinger  at  13.25  Hz  to  control  the  cursor 
horizontal  axis.  Literature  and  in-house  experiments  had  indicated  that,  due  to  the  lateral 
arrangement  of  the  Cyberlink  headband  sensors  and  the  fact  that  the  human  eye  is  a 
bioelectro-chemical  dipole,  typical  lateral  ocular  movements  result  in  pronounced 
transients  in  the  low  frequency  forehead  biopotentials  (1.0  -  6.5  Hz  Brainfmgers). 

Consequently,  lateral  cursor  control  was  re-mapped  to  a  low  frequency  Brainfinger  at 
6.25  Hz  to  make  it  more  intuitive,  leamable  and  reliable.  After  this  change,  operators 
found  lateral  control  simple  to  understand  and  execute:  glance  right  and  the  cursor  moves 
to  the  right;  maintain  stationary  gaze  and  the  cursor  moves  to  the  left. 

Vertical  (Y)  Brainfinger  Re-mapping  -  At  first,  the  low  beta  13.25  Hz  Brainfinger  was 
assigned  to  the  remaining  (Y)  axis;  but  tests  with  dozens  of  naive  and  experienced 
Cyberlink  subjects  in  the  two-axis  maze  task  suggested  that  the  separation  or  independent 
response  between  the  two  motive  signals  (required  for  successful  two-axis  control)  might 
be  improved  if  they  were  spread  further  apart  on  the  BrainBody  spectrum.  After  further 
tests  at  various  control  frequencies  suggested  by  the  literature  search  (see  Hypothesis 
Beta/EMG),  the  vertical  axis  was  mapped  to  a  higher  frequency  beta2  Brainfinger  at  25.5 
Hz  and  the  lateral  axis  was  mapped  to  a  delta  Brainfinger  at  1 .25  Hz. 

In  subsequent  tests  and  studies,  this  revised  Brainfinger  mapping  significantly  enhanced 
two-axis  task  performance  accuracy  and  efficiency.  With  maximum  frequency  spread 
between  the  two  Brainfingers  used  for  control,  the  horizontal/vertical  independence 
required  for  effective  two-axis  cursor  control  was  greatly  improved. 

Revised  Element  #2  (and  4): 

Median  and  Upper/Lower  Quartile  Magnitude  Values  for  Motive  Signal  Characteristic 
and  Basis  for  Auto-Adjust  -  after  the  pre-production  prototype  low-noise 
bioamplifier/interface  (see  section  3.5)  hardware  became  available,  Dr.  Henry  Jex  (an 
expert  consultant  in  manual  control)  met  with  Brain  Actuated  Technologies  personnel  to 
evaluate  the  controller  interface  software  and  discuss  possible  improvements.  The 
addition  of  lead  generation  or  lag  to  the  system  to  stabilize  the  human  operator  input  was 
reconsidered.  Due  to  the  more  stable  cursor  response  dynamics  and  higher  controller  gain 
levels  made  practical  by  the  improved  signal  to  noise  ratio  of  the  new  interface  hardware, 
it  was  decided  that  the  best  approach  would  be  to  eschew  additional  dynamics  to  the 
system,  such  as  lead  compensation  etc.,  but  to  refine  the  auto-adjustment  of  the  interface. 
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Dr.  Jex  suggested  that  this  would  enhance  the  richness  of  the  feedback  so  that  the  human 
operator  can  more  easily  provide  his  or  her  own  lead  generation. 

Towards  this  goal,  an  alternative  statistical  signal  analysis  was  proposed  employing 
median  and  upper  and  lower  quartile  values  as  a  more  accurate  characterization  of 
Brainfmger  magnitude  for  cursor  positioning  and  auto-adjustment  of  controller  gain  and 
bias.  Subsequent  implementation  and  testing  of  this  approach  yielded  undetectable 
improvements  in  cursor  control  and  it  was  eventually  abandoned  pending  further  study. 

Revised  Element  #3: 

Velocity  Cursor  Motion  Formula  -  tests  with  early  training  applications  indicated  that  the 
mono-velocity  (single  speed)  formula  occasionally  produced  more  predictable  cursor 
response  than  the  proportional  formula  in  certain  two-axis  control  contexts,  but  the 
analog,  multi-speed  response  experienced  with  the  proportional  formula  was  lost.  To 
recover  this  advantage,  a  velocity  method  of  cursor  positioning  was  developed.  Cursor 
speed  was  varied  in  accordance  with  control  Brainfmger  magnitude.  Comparison  tests  of 
this  variable  velocity  with  the  earlier  mono-velocity  controller  showed  that  accuracy  of 
two-axis  cursor  positioning  improved  dramatically  due  to  the  increased  degree  of  control 
feedback  provided  by  the  velocity  controller. 

Continuing  tests  of  these  ‘first  generation’  control  paradigm  revisions  yielded  an  unexpected 
result:  while  the  Brainfmger  signal  to  axis  re-mapping  improved  performance  in  all  two-axis 
control  scenarios  and  for  all  cursor  motion  methods,  the  original  proportional  formula  proved  to 
be  the  most  effective  cursor  motion  method  developed  thus  far  in  the  one-axis  pong  task  where 
cursor  movement  had  to  be  synchronized  with  an  incoming  target. 

Also,  a  velocity  controller  incorporating  high  cursor  speed  and  a  small  null  zone  enabled  subjects 
to  complete  the  two  dimensional  maze  task  rapidly,  but  was  found  to  be  excessively  unstable  in 
other  two-axis  settings  more  closely  resembling  the  WINDOWS  desktop  ‘open’  environment. 
Was  the  presence  of  guiding  and  motion-limiting  walls  in  the  maze  a  factor? 

After  these  first  findings,  it  became  clear  that  continued  evolution  of  the  control  paradigm  would 
require  the  development  of  further  training  and  testing  applications  to  model  the  full  range  of 
possible  cursor  performance  settings. 
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3.0  HARDWARE  DEVELOPMENT  UNDER  SBIR  PHASE  II 


3.1  The  Dry  Surface  Electrodes 

Regardless  of  the  application,  the  Cyberlink  Interface's  ease  of  utilization  is  paramount.  The 
primary  interface  between  the  human  and  the  Cyberlink  interface  is  the  sensor  array  that  is 
attached  to  the  subject's  forehead. 

In  the  controlled  laboratory  setting  of  the  SBIR  I  and  SIRE  studies,  the  electrode  measurement 
system  used  hydro-gel  surface  electrodes  for  obtaining  forehead  biopotentials.  The  electrode 
harness  was  noninvasive,  and  relatively  easy  to  connect  to  the  subject.  However,  excessively  dry 
forehead  skin  required  hydration  with  water  while  excessive  oil  on  the  skin  had  to  be  removed 
with  an  alcohol  swab  in  order  to  reduce  surface  electrical  resistance  and  obtain  acceptable 
performance. 

Because  the  electrolyte  on  the  electrodes  was  a  hydro-gel  colloid  it  tended  to  dry  out  when  not  in 
use.  It  was  often  necessary  to  hydrate  each  electrode  before  use.  Even  special  packaging  and 
storage  methods  were  inadequate  to  prolong  shelf  life  beyond  a  dozen  or  so  uses. 

Therefore,  a  practical  dry  electrode  system  needed  to  be  developed  for  improved  durability,  ease 
of  use,  reliability,  and  performance  in  all  operational  settings.  Candidate  electrodes  as  reported  in 
the  literature  were  evaluated  (Prutchi  et  al.,  1993,  Bourland  et.  al.,  1978;  Betts  et  al.,  1976; 

Bergey  et  al.,  1971;  Potter  &  Menke,  1970;  and  Johnson  et.  al.,  1968).  Some  of  these  required 
skin  abrading  and/or  the  application  of  electrolyte  preparations. 

Prototype  stainless  steel  dry  electrodes  had  been  fabricated  by  B.A.T.,  Inc.  just  prior  to  the 
initiation  of  this  SBIR  Phase  II,  and  added  to  a  modified  headband  harness.  Preliminary 
performance  tests  showed  that,  although  they  appeared  to  provide  improved  durability  and  shelf 
life,  required  minimal  skin  preparation  and  exhibited  reduced  D.C.  offset  (and  thus  reduced  input 
amplifier  saturation),  impedance  figures  were  between  70  K  and  1,000  K  ohms,  significantly 
higher  than  those  typically  obtained  with  the  hydro-gel  electrodes. 

High  impedance  electrodes  promote  the  corruption  of  the  forehead  signal  by  noise  and  power  line 
frequencies,  particularly  if  the  length  of  the  electrode  lead  is  excessive.  It  is  possible  to 
incorporate  shielding  in  the  lead  to  decrease  noisy  interference,  however,  the  shield  contributes  to 
distributed  capacitance  and  can  lower  the  input  impedance  to  the  bioamplifier  to  the  point  where 
high  frequency  components  of  the  signal  are  greatly  attenuated. 

This  relatively  high  impedance  may  explain  the  subtle  but  detectable  degradation  in  the  accuracy 
and  "feel"  of  cursor  control  reported  by  experienced  subjects  using  the  stainless  steel  electrodes 
in  one  and  two-axis  Cyberlink  tasks. 
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Consequently,  a  silver  chloride-plated,  carbon-filled  plastic  sensor  system  was  developed  for  this 
Phase  II  effort  by  B.A.T.,  Inc.  working  in  conjunction  with  an  outside  contractor  (Figure  1). 


Carbon-filled  with 
AgCl  plating 


-  forehead  contour  curvature 

rounded  edges  for  comfort 

- - sensor  mounting  snap 


Figure  L  Dry  Electrode  Features 


After  an  iterative  process  of  testing,  redesign,  refabrication,  and  retesting  extending  over  the 
length  of  the  SBIR II  contract,  the  silver  chloride-plated  sensors  were  perfected.  The  finalized 
pre-production  version  exhibits  the  following  characteristics: 

1)  improved  resistance  to  corrosion  and  oxidation, 

2)  a  snap-on  headband  mounting  system  for  easy  removal  for  cleaning  and  maintenance, 

3)  minimal  or  no  skin  preparation  required  in  most  operational  settings, 

4)  forehead  contour  curvature  (horizontally)  for  improved  fit, 

5)  high  durability  and  longevity,  in  some  cases  maintaining  operational  levels  of  performance  for 
several  months  of  continuous  use, 

6)  very  low  impedance  (<  50  K  ohms)  and  noise  levels  typically  obtained  with  little  or  no  skin 
preparation,  and 

7)  performance  equal  to  or  better  than  the  hydro-gel  electrodes  in  direct  comparison  tests. 


3.2  Sensor  Preparation:  Electrolyte  Cream  to  Aloe  Vera  Gel  Moisturizer 

Before  the  development  of  the  Cyberlink  dry  surface  electrodes,  the  hydro-gel  electrodes  that 
were  employed  required  hydration  prior  to  each  use.  Similarly,  the  user’s  forehead  was  often 
prepped  with  tap  water  with  a  moist  cloth  or  eyedropper.  Impedance  values  obtained  when  these 
preparations  were  observed  were  as  much  as  95%  lower  than  when  the  electrodes  were  applied 
with  no  preparation. 

As  the  dry  surface  electrodes  became  operational  under  this  SBIR  II  contract,  preliminary 
performance  studies  showed  that  impedance  levels  obtained  with  no  preparation  were  often  as 
low  as  those  obtained  with  the  hydro-gel  electrodes  with  preparation. 
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Impedance  levels  were  found  to  be  negatively  impacted,  however,  when  the  forehead  skin  was 
excessively  dry  or  oily.  At  such  times  a  conductive  cream  electrolyte  was  found  to  bring 
impedance  back  down  to  optimal  levels  (<  50  K  ohms).  After  several  months  of  use  in  pilot 
tests,  focus  studies,  and  general  Cyberlink  development  applications,  many  users  began  to  report 
the  conductive  cream  made  their  foreheads  feel  “greasy”  and  seemed  to  contribute  to  clogged 
skin  pores  and  occasionally  forehead  skin  blemishes. 

As  a  result,  a  common  EKG  conductive  gel  was  tested.  This  gel,  although  mitigating  the  earlier 
side-effects,  contained  a  salt  which  caused  a  pitting  in  the  silver  chloride  platting  and 
significantly  reduced  the  dry  sensors’  shelf  life.  Some  users  experienced  a  mild  skin  irritation. 

The  third  and  final  solution  was  discovered  when  one  B.A.T.,  Inc.  researcher  ran  out  of  the 
electrolyte  gel  and  tried  aloe  vera  moisturizing  gel  as  a  substitute.  Promising  initial  results  led  to 
more  exhaustive  comparative  testing  which  indicated  that  consistent,  very  low  impedance  levels 
could  be  obtained  with  no  “greasy”  forehead  effect  or  skin  irritation.  It  was  later  discovered  that 
aloe  vera  gel  contains  natural  electrolytes  which  minimize  corrosion  of  the  silver  chloride  sensor 
platting. 


3.3  The  Sensor  Harness 

The  Sensor  Harness  evolved  gradually  over  a  period  of  two  years  as  B.A.T.,  Inc.  and  a 
subcontractor  worked  through  dozens  of  less  than  perfect  versions  to  the  present  fully  operational 
design.  This  finalized  version  embodies  (Figure  2): 

1)  special  touch  proof  connectors  to  the  interface  unit, 

2)  a  composite  jack  and  harness  using  three  potted  plastic  snaps  for  easy  sensor  removal, 

3)  durable  joints  and  connections, 

4)  shielded  high  strength  tensile  wire  in  the  sensor  cable,  and 

5)  very  low  signal  corruption  by  high  frequency  noise. 
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3.4  The  Cloth  Headband 


A  special  cloth  headband  was  developed  by  B.A.T.,  Inc.  and  fabricated  by  an  outside  contractor 
(Figure  2).  It  was  designed  to  be  comfortable,  durable,  and  to  allow  easy  removal  and  reinsertion 
of  the  sensor  harness  for  headband  washing,  and  to  allow  the  sensor  cable  to  lead  from  either  the 
left  or  right  side. 


Cloth  Headband 


Figure  2.  Headband  and  Sensor  Harness 


3.5  The  Interface  Unit 

Pre-Phase  II  studies  had  established  that  a  bioamplifier  with  extremely  high  sensitivity  needed  to 
be  developed  before  the  interface  could  realize  its  full  performance  potential.  Sufficient 
sensitivity  to  amplify  and  faithfully  represent  very  small  signals  resulting  from  both  brain  and 
subtle  muscle  activity  was  required.  These  signals  are  of  highest  interest  because  the  Cyberlink 
interface's  prime  focus  is  to  approach  an  ideal  situation  where  control  is  essentially  "thought 
actuated".  These  small  signals  can  range  from  as  low  as  0.5  microvolts  to  strong  frontal-lobe 
alpha  waves  of  15  to  50  microvolts.  The  obvious  problem  with  acquiring  a  signal  that  is  in  the 
order  of  0.5  microvolts  is  the  noise  floor  of  the  amplifier  circuitry.  Both  active  and  passive 
circuit  elements  contribute  to  the  production  of  noise  and  reduce  the  signal-to-noise-ratio  of  the 
system. 

Higher  controller  gain  was  expected  to  allow  less  control  signal  filtering  in  the  software,  more 
efficacious  feedback  to  the  user,  and  therefore  better  cursor  control.  Nevertheless,  without  a 
reduction  in  perceived  system  noise,  operators  could  not  successfully  operate  at  higher  controller 
gain. 

To  compound  the  problem,  while  the  human  operator  is  producing  these  very  subtle  signals, 
spontaneous  muscle  activity  on  the  forehead  often  produces  very  large  signals  that  can  have 
voltage  peaks  in  the  millivolt  range.  Some  of  these  signals  are  very  low  frequency  and  cause  the 
integrated  circuit  amplifiers  to  saturate  at  the  power  supply  limits  for  a  period  of  time,  thus 
causing  the  loss  of  information  until  the  circuits  recover.  On  the  other  hand,  these  very  large 
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signals  can  also  be  of  interest  and  be  utilized  by  the  Cyberlink  interface  as  part  of  its  control 
mechanism. 

The  bioamplifier  had  to  incorporate  an  improved  dynamic  range,  e.g.,  high  sensitivity  to  detect 
the  very  small  signals  while  maintaining  the  capacity  to  process  very  large  inputs  without  causing 
components  to  saturate. 

The  prototype  Cyberlink  interface  system  functioned  adequately  with  all  the  signal  decoding 
performed  by  the  host  computer.  However,  decoding  tasks  are  time  sensitive  and  prevent  the 
running  of  third  party  software  with  the  Cyberlink  interface  software.  A  microprocessor 
incorporated  into  the  bioamplifier  was  necessary  to  output  a  EEG/EMG/EOG  data  stream  to  the 
host  computer. 

An  in-house  electrical  engineer  was  hired  for  this  effort  with  the  necessary  expertise  to  address 
these  and  other  challenges.  Over  a  period  of  16  months,  the  final  preproduction  prototype 
Bioamplifier/ Analog-to-Digital-Converter/Microcontroller  Interface  Unit  (the  ‘interface  unit’) 
was  developed  by  B.A.T,  Inc.  and  a  subcontractor-manufacturer.  It  embodies  a  two-layer,  pin- 
through-hole  and  surface  mount  printed  circuit  board,  a  3.3  volt  battery  power  supply  with 
optical  isolation  and  remote  software  control,  durable  plastic  housing  with  low  desktop  ‘profile’, 
copper/silver  spray  shielding  of  housing  interior  to  reduce  electromagnetic  susceptibility,  a 
greatly  reduced  noise  floor,  increased  signal  resolution,  and  improved  analog  signal  filtering 
characteristics  (Figure  3). 


Figure  3.  Interface  Unit 


The  interface  unit  also  has  three  separate  channels  for  EEG,  EOG,  and  EMG  amplification, 
filtering,  and  processing,  reverse  voltage  protection  on  the  circuit  board,  microprocessor  (PIC 
chip)  conversion  of  signal  data  to  an  RS232  (serial)  port-compatible  bi-directional  data  string, 
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special  connectors  for  ‘daisy-chaining’  with  a  second  interface  unit  for  two-player  applications,  a 
power  supply  adapter  for  either  internal  or  external  battery  power,  and  full  compliance  with  FCC 
Class  B  standards  and  European  Community  (CE)  residential/light  industrial  standards  for 
emissions  and  external  susceptibility  to  high  frequency  radiation. 


Interface  Unit  Specifications: 

•  Analog  Gain:  500,000 

•  Noise  At  Front  End:  Less  than  0.3  micro-volts 

•  Analog  Bandwidth:  0.2  Hz  to  3,000  Hz 

•  Analog  to  Digital  Conversion:  6  Channels,  12  Bit  Accuracy 

•  Power  supply:  2  AA  Batteries  (included) 

•  Isolation:  2,500  Volts 

•  PC  connection:  RS232  serial  port  connection 

•  Dimensions:  190mm  x  100mm  x  40mm 

•  Weight:  13.5  oz. 

•  reverse  voltage  protection 

•  approved  by  the  FCC  and  CE 
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4.0  SOFTWARE  DEVELOPMENT  UNDER  SBIR  PHASE  H 


4.1  The  WINDOWS  Visual  C++  Server  and  Launcher 

To  use  the  Cyberlink  successfully  in  operational  settings,  its  forehead  signal  command  data, 
converted  into  mouse  and  keyboard  inputs,  had  to  be  made  accessible  to  third  party  WINDOWS 
software  applications  running  concurrently  with  any  necessary  Cyberlink  control,  adjustment,  or 
display  utilities.  Tasking  some  of  the  signal  conditioning  and  analysis  to  the  microcontroller  in 
the  interface  box  was  the  first  prerequisite  to  accomplishing  this.  Object-oriented  code 
conversion  and  porting  of  the  prototype  Basic  DOS  programs  to  C++  and  WINDOWS  was  the 
second. 

First,  a  Visual  C++  server  program  was  created  with  a  synthetic  data  object  allowing  testing  and 
revision  of  the  various  signal  decoding  and  control  methods  as  the  hardware  was  being  perfected. 
This  synthetic  data  object  consisted  of  a  sum-of-sine- waves  file  that  approximated  the  expected 
signal  output  of  the  Cyberlink  interface  hardware.  Next,  an  object-oriented  EMG  class  divided 
into  two  units  of  functionality,  the  ‘synthetic’  and  ‘real’  EMG  events,  was  created. 

A  ‘test  server’,  a  WINDOWS  VBX,  and  Dynamic  Link  Libraries  (DLL’s)  were  constructed, 
documented  and  supplied  to  consultants  to  allow  off-site  development  and  testing  of  new 
WINDOWS  training  programs  without  the  need  for  distributing  the  early  versions  of  the 
hardware. 

User  interface  software  utilities  were  developed.  A  WINDOWS  Cyberlink  launcher  program 
was  designed  to: 

1)  initiate  a  hardware-user  interface  continuity  check  which  determined  connection 
integrity  at  the  Headband  and  RS232  port  and  displayed  appropriate  error  boxes  with  corrective 
procedures  to  the  user, 

2)  provide  a  continuous  system  integrity  test  alerting  the  user  with  an  appropriate  error 
box  whenever  the  connection  to  the  interface  unit  was  disrupted  or  lost. 

3)  create  a  user  log-in  screen  to  provide  an  ID  under  which  to  store  the  user's  interface 
formatting  options,  calibration  and  adjustment  values  in  the  server's  INI  files, 

4)  provide  an  expository  graphical  user  interface  with  a  composite  Brainfmger,  EEG, 
EMG  and  EOG  display  panel  in  clear  and  engaging  graphics  running  in  a  window  concurrently 
with  other  Cyberlink  training  applications  to  afford  timely  and  intuitive  user  feedback  for  quick 
understanding  and  mastery  of  Cyberlink  control  (see  Figure  4), 
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Figure  4:  Cyberlink  Signal  WINDOWS  “ Display  Panel ” 
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and  6)  create  a  quick,  easily  accessed  user  interface  self-adjustment  program  modeled 
after  familiar  WINDOWS  joy  stick  or  mouse  adjustment  routines  (see  Figure  6). 
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Figure  6:  Cyberlink  WINDOWS  “Joystick  Adjust”  Utility 


4.2  Development  of  User  Calibration  and  Interface  Adjustment 
Calibration  (Initial  Adjustments) 

Method  One:  Brain  Billiards  -  The  user  adjustment  interface  employed  in  the  Phase  I  SBIR 
experiments  had  encouraged  the  subjects  to  explore  a  previously  untested  range  of  cursor  gain 
and  offset  adjustment  settings  with  positive  effects  on  task  performance  and  efficiency.  New 
tasks  were  developed  for  initial  SBIR  Phase  II  experiments  to  further  explore  this  enlarged 
adjustment  space  and  to  provide  data  for  the  design  of  an  improved  user-interface  calibration  task 
in  which  performance  error  was  converted  (through  a  three-dimensional  matrix)  into  optimal 
initial  adjustment  values. 
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These  results  culminated  in  the  development  of  a  WINDOWS  application  called  “Brain 
Billiards”.  Brain  Billiards  combined  a  simple  training  task  requiring  sequential  control  often 
Brainfmger  magnitudes  (similar  to  “SlowBall”  from  the  original  DOS  programs)  with  the 
calibration  function  (see  Figure  7).  This  utility  was  added  to  the  WINDOWS  Cyberlink  launcher 
program  and  subsequent  tests  demonstrated  improved  performance  in  two-axis  control  tasks 
among  subjects  whose  initial  interface  adjustment  values  were  established  by  the  Brain  Billiards 
matrix. 
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Method  Two:  Quick  Start  Personality  Type  -  Studies  were  conducted  to  verify  anecdotal 
evidence  indicating  a  strong  correlation  between  personality  type  and  ambient  or  baseline  power 
in  the  BrainBody  signal.  Signal  data  and  personality  questionnaires  were  collected  for  65 
subjects  (including  40  students  at  a  local  high  school).  Self-reported  Type  A  personality 
operators  (assertive,  competitive,  results-oriented)  were  found  to  have  significantly  higher 
baseline  BrainBody  signal  levels  than  type  B  (low-key,  cooperative,  process-oriented). 

A  new  initial  gain  and  offset  calibration  program  based  on  self-reported  personality  type  was 
implemented.  Subsequent  evaluation  with  one  and  two-axis  training  applications  indicated  this 
method  of  determining  initial  user  adjustments  to  be  as  effective  as  the  higher  resolution 


adjustment  matrix  supplied  by  the  Brain  Billiards  calibration  task.  It  was  added  to  the  Launcher 
as  an  alternative  calibration  method  called  “Quick  Start”. 

Automatic,  On-Line  Continuous  Interface  Adjustment 

Experience-based  revision  of  a  continuous  auto-adjustment  routine  continued  throughout  the 
software  development  phase.  This  behind-the-scenes  program  made  continuous  automatic 
corrections  to  the  initial  calibrations  by  adjusting  the  user-interface  values  effecting  system  gain 
and  cursor  offset  for  each  Brainfinger.  The  auto-adjustments  were  based  on  sliding  window 
averages  of  magnitude  means  and  standard  deviations  for  each  Brainfinger. 

Ultimately,  with  the  improved  functionality  of  both  the  interface  hardware  and  the  control 
paradigm,  it  was  determined  that  little  was  gained  with  continuous  auto-adjustment.  Often  cursor 
control  was  negatively  affected  and  operators  reported  that  they  were  working  at  cross  purposes 
with  the  auto-adjustment  routine  as  it  continuously  corrected  and  compensated  for  their 
intentional  time-averaged  control  inputs  as  well  as  for  unintentional  input  signal  dynamics. 
Eventually,  after  several  revisions  failed  to  achieve  the  desired  performance,  the  continuous  auto¬ 
adjust  approach  was  abandoned. 

User-Selected  Adjustments 

Joystick  Adjust:  The  joystick  adjust  utility  was  developed  as  an  interactive  display  of  Cyberlink 
two-axis  control  with  click-and-drag  scrollbars  to  implement  user  updates  to  the  settings  for  gain 
and  cursor  offset.  It  was  later  refined  to  include  separate  control  algorithm  types,  each  providing 
different  operational  control  characteristics  (proportional,  mono-velocity,  velocity,  etc.)  from  the 
same  BrainBody  signal  source.  A  simple  Brainfinger-to-axis-of-control  mapping  utility  also  was 
added  to  facilitate  experiments  with  alternative  configurations  for  this  element  of  the  control 
paradigm.  As  with  the  display  panel  and  other  utilities,  joystick  adjust  could  run  concurrently 
with  any  other  Cyberlink  WINDOWS  utility  or  application. 


4.3  Initial  System  Integration  and  Documentation 

After  the  calibration  and  adjustment  programs  became  fully  operational  and  were  tested  and 
refined,  they  were  integrated  into  the  launcher  program.  A  proper  syntax  was  designed  and 
implemented  in  the  launcher  to:  1)  register  the  user,  2)  validate  continuity  of  the  sensors, 
interface  unit  to  computer  connection,  and  interface  unit  battery  power  status,  3)  load  the  server, 
4)  launch  the  selected  training  applications  with  the  appropriate  user  environment  (i.e.  user 
calibration,  adjustment  and  format  settings)  if  the  user  is  previously  registered  with  the  launcher, 
or  5)  launch  the  Brain  Billiards  (or  Quick  Start)  calibration  routine  if  the  user  had  not  previously 
registered  with  the  launcher. 

Integrated  testing  of  the  functionally  stable  interface  hardware  and  server,  the  launcher  programs, 
the  control  paradigm(s),  and  the  first  WINDOWS  training  programs  with  actual  biological 
signals  from  the  interface  (as  opposed  to  the  computer-simulated  signal  from  the  test  server) 
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became  possible.  Preliminary  results  of  these  initial  system  tests  gave  rise  to  some  of  the  ‘first 
generation’  control  paradigm  design  changes  discussed  in  section  2.4  and  also  guided  the 
evolution  of  the  operational  characteristics  of  the  training  programs. 

Next  an  installation  program  for  the  WINDOWS  server  and  launcher  software,  as  well  as  the 
training/test  applications,  was  created.  Due  to  the  increased  complexity  of  programs  running  in 
the  WINDOWS  environment  there  was  a  need  for  special  software  to  install  the  programs  onto 
the  computer.  Attention  was  given  to  insure  that  correct  versions  of  the  VBX  and  DLL  and 
various  other  program  elements  were  copied  to  the  appropriate  locations  in  the  operating  system. 

Due  to  the  size  of  the  software  programs,  an  alternative  to  the  floppy  disk  storage  medium  was 
developed  and  perfected  so  that  the  Cyberlink  WINDOWS  software  could  be  ported  from  one 
computer  to  another  via  CD/ROM. 

Integration  of  all  system-level  code  (i.e.  server,  VBX,  DLL,  launcher  utilities,  etc.)  into  standard 
installation  programs  and  testing  with  the  training  applications  continued  throughout  the  software 
development  phase.  The  installation  procedure  for  the  system  software  modules  and  the  training 
applications  was  updated  and  new  install  disks  were  prepared  as  the  software  continued  to 
evolve. 

Documentation 

As  a  prerequisite  to  off-site  beta-test  release,  software  documentation  in  the  form  of  on-line  help 
screens  and  'quick-tips'  was  added  to  the  WINDOWS  launcher  and  other  Cyberlink  applications. 

In  addition,  a  50  page  Cyberlink  User  Manual  and  a  12  minute  tutorial  video  were  written.  This 
material  was  regularly  revised  and  updated  throughout  the  SBIR  Phase  II  effort  to  keep  it  current 
with  interface  system  hardware  and  software  developments. 


4.4  WINDOWS  Training/Test  Applications 

Development  of  training  and  evaluation  software  continued  throughout  the  latter  half  of  the  SBIR 
Phase  II  effort.  Independent  contractors  were  employed  to  design  graphics,  create  graphics 
engines  and  develop  new  two-axis  plus  ‘click’  control  test  environments  to  more  accurately 
model  the  full  operational  functionality  of  the  WINDOWS  mouse. 

The  most  notable  of  the  new  training/test  applications  involved  target  acquisition  by  two-axis 
Brainfinger  cursor  positioning,  ‘click’  selection  and  ‘drag’  by  EMG  spike  inputs,  and  an  EMG- 
spike  target  release  over  a  receiving  area.  This  application  closely  modeled  the  cursor 
functionality  and  control  issues  in  the  WINDOWS  desktop  environment  and  became  very  useful 
in  providing  data  to  inform  and  evaluate  ongoing  control  paradigm  developments. 

A  first-person,  multi-media,  three-dimensional  maze  application  was  also  developed  in  which 
lateral  cursor  commands  translated  to  rotation  in  the  virtual  environment  and  vertical  commands 
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translated  to  forward  and  reverse  motion.  This  application  exposed  many  controller  stability 
issues,  eventually  resulting  in  the  development  of  several  high  stability  control  paradigms. 

Further  enhancement  of  the  interface  was  done  by  adding  the  initial  capability  for  two-user 
control  in  the  WINDOWS  environment  and  a  new  dual-player  ‘pong’  training  application  was 
developed. 

4.5  WINDOWS  3.1  Cyberlink  Mouse  Driver  Prototype  -  “Hypermouse” 

As  the  Cyberlink  software  matured,  a  parallel  effort  was  begun  to  develop  a  WINDOWS  Mouse 
driver  allowing  simultaneous  cursor  control  of  third-party  WINDOWS  software  applications  by  a 
standard  mouse  and  the  Cyberlink  controller.  A  device  driver  specialist  was  contracted  from 
out-of-state  as  it  was  not  possible  to  find  a  local  consultant  with  the  required  expertise  to  develop 
software  of  this  complexity.  Shortly  after  his  arrival  at  B.A.T.,  Inc,  the  consultant  identified 
problems  in  the  microprocessor  protocol  that  needed  to  be  corrected  before  driver 
implementation  could  be  completed. 

After  three  weeks  of  very  intensive  work,  the  consultant,  working  with  B.A.T.  personnel, 
completed  the  prototype  WINDOWS  3.1  mouse  driver.  The  driver,  initially  dubbed 
“Hypermouse”,  contained  the  necessary  functionality  to  service,  simultaneously,  standard  mouse 
inputs  and  decode  two  user-selected  Brainfingers  from  the  Cyberlink  interface  for  x-y  cursor 
control  by  issuing  the  appropriate  mouse  movement  commands  in  accordance  with  the  selected 
Cyberlink  control  paradigm.  In  addition,  the  driver  computed  the  click  events  from  the 
Cyberlink  interface  and  performed  the  x/y/EMG  auto-adjustment. 

Due  to  the  elegant,  computationally  inexpensive  nature  of  the  Cyberlink  decoding  process  all  of 
the  driver  capabilities  could  be  accomplished  with  a  driver  that  occupied  less  than  10  kilobytes  of 
memory. 

A  Hypermouse  parameter  block  was  designed  so  that  the  user  could  selectively  tum  on  Cyberlink 
driven  mouse  actions;  some  or  all  at  the  same  time.  For  example,  the  user  could  elect  to  activate 
only  the  EMG-click  or  the  cursor  lateral  control  at  one  time.  Preliminary  tests  with  the  Cyberlink 
EMG-click  and  the  standard  mouse  x-y  cursor  control  suggested  this  system  combination  might 
exhibit  advantages  in  operational  settings  where  short  discrete  response  reaction  times  are 
critical. 

A  Hypermouse  interface  setup  routine  enabled  the  user  to  launch  the  Hypermouse  with  their 
necessary  .ini  file  information  (including  interface  formatting,  adjustments  and  calibrations  made 
in  the  WINDOWS  Cyberlink  launcher  utilities)  transferred  to  the  Hypermouse  parameter  block. 
The  user  also  selected  which  Cyberlink-driven  mouse  commands  were  to  be  active  with  the 
Hypermouse  setup  routine. 

Preliminary  tests  of  the  Hypermouse  x-y  control  indicated  that  the  training  applications  and 
interface  calibration/adjust  routines  were  crucial  to  prepare  the  user  prior  to  launching  into  the 
WINDOWS  desktop  environment  with  the  Hypermouse.  In  this  way,  interface  values  optimizing 
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the  user’s  cursor  control  were  set  in  the  Hypermouse  parameter  block.  In  the  course  of  these 
tests,  the  value  of  the  WINDOWS  architecture  and  the  Cyberlink  training  applications  and  other 
Cyberlink  supporting  software  utilities  became  more  fully  appreciated. 

4.6  First  Integrated  System  Assessment 

As  the  prototype  driver;  the  headband  and  interface  hardware;  the  user  manual  and  other 
documentation;  the  server;  the  launcher  formatting,  adjustment,  and  training  utilities;  and  the 
new  training/test  applications  further  matured  they  began  to  mesh  together  into  a  fully  functional, 
integrated  system  ready  for  beta  testing. 

At  this  stage  in  the  Cyberlink  development  effort,  more  information  in  the  form  of  interface 
signal  and  task  performance  data,  debriefing  questionnaires,  and  anecdotal  verbal  feedback  was 
essential  for  further  progress.  Throughout  the  latter  half  of  the  SBIR  Phase  II  effort,  extensive 
tests  and  focus  studies  with  a  large  and  varied  subject  population  were  conducted  in-house;  at 
assistive  technology  and  computer  electronics  conventions,  conferences,  and  trade  shows;  at 
local  public  schools  and  universities;  and  at  beta  sites  in  private  homes  and  institutions  all  over 
the  US,  England,  Europe,  and  Canada. 

Overall,  the  results  indicated  that  system  and  operator  performance  depended  on  four  primary 
factors: 

•  operational  setting  -  ambient  room  noise,  distractions,  and  stressful  influences  affecting 
concentration;  temperature  conditions  promoting  excessive  forehead  sweat  or  the  presence  of 
oil-based  makeup  on  the  forehead  that  reduce  the  efficacy  of  the  user’s  connection  to  and 
control  of  the  interface;  and  other  general  environmental  conditions. 

•  operators’  functional  limitations  -  users  with  traumatic  brain  injuries,  Cerebral  Palsy  and 
other  conditions  limiting  facial  muscle  functionality,  sight  impairments,  autism,  children 
under  five  years  of  age  or  others  with  attention  deficit/hyperactivity  disorder,  and  ‘severe 
chronic  skepticism’  or  a  personal  temperament  reflecting  an  attitude  of  distrust  of  the  novel 
and  unfamiliar.  All  of  above  may  limit  ease  of  Cyberlink  interface  learning,  calibration  and 
use. 

•  length  of  training  -  Basic  one-axis  ‘pong’  and  two-axis  ‘maze’  control  was  often  realized  in 
the  first  10  minutes  of  training.  However,  many  users  found  that  two-axis  plus  ‘click’ 
WINDOWS  desktop-type  applications  required  training  of  up  to  five  or  six  30-minute 
sessions  before  asymptotic  performance  was  attained. 

•  operator’s  ability  to  achieve  independent  signal  control  -  Approximately  one  out  of  five 
people  tested  exhibited  an  innate  ability  to  raise  selectively  the  signal  input  mapped  to 
vertical  cursor  positioning  without  simultaneously  and  involuntarily  raising  their  lateral  input 
signal  or  unintentionally  issuing  an  EMG-click  command.  Two  out  of  five  people  tested 
showed  that  they  could  readily  develop  this  crucial  Cyberlink  performance  skill  in  the  course 
of  training  by  reducing  the  EMG  or  forehead  muscle  component  of  their  vertical  axis  control 
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technique.  The  remaining  two  out  of  five  appeared  to  be  limited  in  their  ability  to  develop 
sufficient  signal  independence  within  a  practical  training  period  (six  30-minute  sessions)  to 
achieve  any  substantial  degree  of  two-axis  plus  click  control 

4.7  Second  Generation  Control  Paradigm  Development 

The  most  successful  control  paradigm  used  in  the  first  system  assessment  involved  an  EMG- 
spike  discrete  (click)  command  and  velocity  continuous  cursor  x-y  positioning  based  on  the  time- 
averaged  magnitudes  of  a  delta  and  beta2  Brainfinger,  respectively.  This  paradigm,  an  extension 
of  the  original  pre-SBER  II  control  model,  came  to  be  known  as  “The  Classic”  controller.  The 
assessment  results  pointed  to  the  need,  and  suggested  an  approach,  for  additional  control 
paradigm  development,  testing,  refinement,  and  finally  integration  into  a  “best-of  ’  synthesis  to 
create  a  universal  interface  -  one  able  to  be  learned  and  applied  by  a  majority  of  users  in  a  broad 
range  of  operational  settings  and  applications. 

First,  the  following  controller  performance  criteria  set  was  developed: 

1)  Separation  -  degree  of  independence  of  x  axis  cursor  control  inputs  from  y  axis  cursor  control 
inputs.  Also,  the  degree  of  independence  of  y  axis  inputs  from  EMG-cIick  commands. 

2)  Stability  -  degree  to  which  the  cursor  retains  its  position  in  the  absence  of  intentional 
continuous  control  inputs.  Does  it  hover  in  the  same  basic  area  of  the  display  or  wander 
chaotically  between  control  inputs? 

3)  Accuracy  (High  Resolution)  -  the  degree  to  which  small,  volitional,  easily  executed  inputs 
will  reliably  direct  the  cursor  short  distances  to  achieve  intended  positions  and  targets  occupying 
small  screen  areas  (i.e.,  <  32  x  32  pixels). 

4)  Rapidity  (Low  Resolution)  -  the  degree  to  which  larger,  volitional,  easily  executed  inputs  will 
rapidly  direct  the  cursor  large  distances  to  intended  general  screen  areas  (i.e.,  >  32  x  32  pixels)  in 
a  timely  fashion. 

5)  Learnability  (Intuitiveness)  -  the  degree  to  which  all  of  the  above  can  be  learned  and  achieved 
by  the  user  after  a  minimum  of  explanation,  adjustment  and  training. 

Armed  with  the  performance  criteria,  the  system  assessment  data,  and  the  earlier  literature  review 
and  experimental  data,  a  team  of  in-house  and  independent  contractors  worked  on  the  problem  of 
developing  a  second  generation  of  control  paradigm  designs.  Innovative  solutions  were 
encouraged.  No  idea  was  rejected  on  the  basis  of  being  too  ‘far-out.’ 

Within  a  few  weeks,  nearly  a  dozen  new  control  paradigms  were  developed  and,  along  with  ‘The 
Classic’  controller,  were  subjected  to  evaluation.  The  most  promising,  those  preferred  by  a 
significant  percentage  of  the  evaluators,  are  shown  below  with  their  grades  in  each  of  the 
performance  criteria  given  on  an  ascending  scale  from  one  to  five. 
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Evaluation  of ‘Second  Generation’  Control  Paradigms 


Paradigm 

Design  Elements 

separation 

accuracy 

jj/VctlUaLl 

stability 

uu 

rapidity 

leamability 

TOTAL 

The  Classic 

1) X=  1.25  Hz,  Y  =  25.25  Hz 

2)  power  i  (time  averaged  magnitudes) 

3)  velocity  (increases  with  AMag) 

4)  gain  and  cursor  offset  adjustments 

5)  stationary  in  motive  signal  null-zone 

6)  EMG,  EOG,  or  selected  Brainfinger 
spike  activates  ‘mouse5  click  command 

2-4 

2-4 

2-3 

3-5 

2-4 

11  -20 

Comments: 
very  user 
dependent 
(see  section 

4.6) 

Etch-a-Sketch 

1)  X  =  1.25  Hz,  Y  -  1.25  Hz  (not 
simultaneous) 

2)  power  j  (time  averaged  magnitudes) 

3)  mono- velocity  (fixed- speed  threshold) 

4)  gain  and  cursor  offset  adjustments 

5)  stationary  in  null-zone  and  on  inactive 
axis 

6)  EMG  single  click  toggles  active  axis 
of  motion;  double-click  is  mouse  click 

5 

3.5 

4.5 

3 

3.5 

19.5 

Comments: 
very  stable 
but  click¬ 
intensive, 
fatiguing, 

“not 

elegant” 

Tank  Stick 

1) X  =  fmae  1.25  Hzl 

[mag  25.25  Hz] 

Y  -  [mag  1.25  Hz]  +  [mag  25.25  Hz] 

2)  power!  (time  averaged  magnitudes) 

3)  proportional 

4)  gain  and  cursor  speed  adjustments 

5)  stationary  in  null-zone 

6)  EMG  spike  ‘mouse’  click  command 

2 

2-3 

2 

3 

2-3 

11  -  13 

Comments: 
“fun  to  use,” 
“not 

practical” 

Flight  Stick 

1)  X  =  Imas  1.25  Hzl 

[mag  25.5  Hz] 

Y  —  constant  upward  motion  plus 
‘wrap’  feature  (cursor  lowest  vertical 
screen  position  follows  highest) 

2)  power!  (time  averaged  magnitudes) 

3)  proportional 

4)  gain  and  cursor  speed  adjustments 

5)  constant  upward  motion  even  when 
motive  signal  in  null-zone 

6)  EMG  spike  ‘mouse’  click  command 

4 

1.5 

1.5 

2.5 

2 

11.5 

Comments: 
“fun  to  use,” 
“not  practical” 

Two  additional  control  paradigms  were  conceived  during  the  second  generation  brainstorming, 
but  were  not  implemented  until  the  subsequent  completion  of  the  third  party  WINDOWS  95 
Mouse  driver: 

1)  EOG  Look  Left/Right  -  a  lateral  control  method  using  the  shape  rather  than  the  power  of  the 
motive  signal  to  determine  the  direction  of  motion  on  the  x  axis.  Based  on  a  left-right  phase 
reversal  phenomenon  observed  in  the  EOG  signal,  corresponding  to  direction  of  lateral  eye 
motion,  a  direction-specific  horizontal  axis  control  method  was  developed.  A  sharp  leftward 
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glance  initiates  leftward  cursor  motion  and  a  sharp  rightward  glance  initiates  rightward  cursor 
motion.  Cursor  velocity  is  modulated  by  another  aspect  of  the  EOG  waveform. 


This  method  is  still  under  evaluation  at  the  time  of  this  writing,  although  several  beta  sites  have 
reported  it  to  be  their  preferred  x  axis  control  method.  Preliminary  evaluation  of  the  current 
embodiment  indicates  high  grades  in  separation  and  stability  but  somewhat  lower  performance  in 
areas  of  accuracy  and  rapidity. 

Small  targets  are  difficult  to  acquire  without  repeated  passes  on  the  lateral  axis  because  the 
cursor  is  difficult  to  move  short  distances  without  ‘overshooting’.  Distant  screen  locations  are 
difficult  to  attain  in  a  timely  fashion  because  the  user  must  wait  until  the  “ready”  cue  indicates 
the  EOG  waveform  has  returned  to  a  state  of  stasis  before  further  control  inputs  can  be  issued. 

2)  The  Cogitator  -  an  analysis  of  signal  ‘chaos’  or  time-averaged  degree  of  variability,  rather  than 
power,  for  cursor  vertical  motion.  The  preliminary  design,  based  on  an  in-house  study  of  35  Hz 
and  40  Hz  gamma  Brainfinger  magnitudes  over  a  fixed-period  sliding  sample  window  during  the 
performance  of  Raven  Progressive  Matrices,  shows  promise  but  a  more  practical  means  of 
evoking  the  motive  signal  characteristic  needs  to  be  developed. 


4.8  Building  a  Better  Mouse:  WINDOWS  95  Mouse  Driver  -  C.A.T. 

The  WINDOWS  3.1  Hypermouse  driver  was  an  excellent  proof-of-concept  prototype  and  served 
as  the  development  model  for  the  more  sophisticated  WINDOWS  95  Cyberlink  Mouse  Driver 
with  expanded  capabilities. 

A  devise  driver  specialist  was  contracted  to  work  on-site  with  the  B.A.T.,  Inc.  software 
development  team  responsible  for  the  design  and  implementation  of  the  second  generation 
control  paradigms.  The  new  driver  and  support  software  was  dubbed  the  “C.A.T.”,  Cyberlink 
Actuated  Tracker.  During  the  final  stages  of  the  SBIR II  software  development,  the  functionality 
was  expanded  to  include: 

•  A  C.A.T.  monitor  program  to  provide  user  discrete  command  feedback  in  the  form  of  visual 
cues  in  the  WINDOWS  task  bar  and  short  WAV  file  auditory  cues. 

•  A  launcher/exit  program  affording  access  to  an  expanded  parameter  block  to  register  the  user 
with  the  appropriate  interface  adjustments  and  formatting;  launch  a  C.A.T.  set-up  program  in 
which  the  user  selected  the  control  paradigm  and  had  access  to  a  joystick-type  x-y  and 
discrete  gesture  adjustment  and  mapping  routine;  launch  the  selected  WINDOWS  95 
application;  start  the  C.A.T.  driver  and  accompanying  C.A.T.  monitor  program;  and  provide 
a  direct  C.A.T.  exit  which  simultaneously  turned  off  the  driver. 

•  A  scanning  option  with  EMG-click  select  for  full  hands-free  navigation  and  control  of  the 
C.A.T.  launcher/exit  and  setup  routines. 
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To  provide  comprehensive  mouse  functionality,  refinements  were  made  to  the  EMG-click  event 
and  additional  discrete  gesture  commands  were  created  as  more  discriminating  EMG  waveform 
analysis  algorithms  were  developed. 

Since  gestures  provide  more  than  button  commands,  the  C.A.T.  control  functions  were  expanded 
to  include  a  facial  gesture  activated  cursor  speed/resolution  change.  This  toggle  function 
integrated  a  low  speed,  high  resolution  mode  and  a  high  speed,  low  resolution  mode  into  a  single 
control  paradigm,  thus  enhancing  both  accuracy  and  rapidity  of  performance. 

To  provide  an  easy  method  to  temporarily  deactivate  the  Cyberlink  control  of  the  mouse, 
restoring  full  control  to  the  manual  mouse  without  turning  off  the  C.A.T.  driver,  a  C.A.T.  “sleep” 
mode  was  created.  The  additional  discrete  commands  were  default  mapped  to  the  expanded 
control  functions  but  could  be  reformatted  by  the  user. 


C.A.T.  Default  Gesture  Mapping 


GESTURE 

CHARACTER 

ACTION(S) 

MONITOR  MESSAGE 

1)  Single  Eyebrow  Lift 

Default  =  none 
user-selectable 

Left  Mouse  Button  Click  & 
briefly  stops  cursor  motion. 

“Left  Click” 

2)  Double  Eyebrow  Lift 

u 

Double  Click  &  stops  cursor 

“Left  Double  Click” 

3)  Triple  Eyebrow  Lift 

u 

Click  and  Drag  &  stops  cursor 

“Left  Drag” 

4)  Jaw  Close 

6t 

Switch  Cursor  Speed  &  Resolution 

“Jaw  Close” 

5)  Leftward  Glance 

a 

Briefly  stops  cursor  motion  &  when 
followed  by  Jaw  Close 
"  toggles  CAT  “sleep/awake”  mode 

“Look  Left” 

* 

6)  Rightward  Glance 

u 

Moves  Cursor  to  Right 

None  (unless  user- 
formatted  with  a  discrete 
command  Action) 

A  special  suite  of  two-axis  plus  click  control  training  applications  was  compiled  under  a 
Cyberlink  program  called  “Trainer”  to  allow  learning  of  Cyberlink  control  skills  and  interface 
formatting  and  adjustment  in  a  less  demanding  environment  prior  to  launching  the  C.A.T.  in  the 
more  intense  WINDOWS  desktop.  Adjustments  and  formatting  made  in  the  Trainer 
automatically  carried  over  to  the  C.A.T.  Programs. 

Once  fully  operational,  the  C.A.T.  driver  and  launcher/setup  programs  were  added  to  the 
Cyberlink  Install  disks  and  were  thoroughly  tested  in-house  and  shipped  to  all  beta  sites. 
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The  final  phases  of  the  SBIR II  effort  comprised  an  exhaustive  iterative  process  of  beta  testing, 
redesign,  implementation,  retesting,  redesign  and  reimplementation. 

During  the  final  performance  evaluation  of  the  SBIR  Phase  II  contract,  a  new  control  paradigm 
was  designed  and  added  to  the  C.A.T.  and  other  Cyberlink  programs: 

EMG  Vertical  Control  -  a  modification  of  The  Classic  controller,  mapping  EMG  power  to 
continuous  vertical  cursor  velocity.  Unintentional  EMG-clicks  are  avoided  because  easily 
controlled,  very  subtle  EMG  inputs  (well  below  the  click  threshold)  reliably  and  accurately  raise 
the  cursor. 

This  development  appears  to  improve  The  Classic’s  performance  in  all  areas,  particularly 
separation  and  accuracy.  A  synthesis  of  this  paradigm  with  elements  from  several  others 
provided  the  controller  used  in  the  final  C.A.T.  vs.  Mouse  comparative  performance  study. 
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5.0  FINAL  CYBERLINK  INTERFACE  PERFORMANCE  TEST:  C.A.T.  vs.  MOUSE 


5.1  Experimental  Method 

Subjects  -  three  men  and  two  women,  ages  ranging  from  26  to  53  years,  with  a  mean  of  39.6 
years,  participated  in  the  experiment.  All  reported  normal  or  corrected-to-normal  vision.  Two 
had  previous  experience  with  the  Cyberlink  interface  and  three  were  naive.  One  subject  was  left- 
handed. 

Materials  -  1)  Input  device:  Integrated  Cyberlink  hardware  ‘beta  test’  system, 

training  and  adjustment/formatting  software  utilities,  and  the  WINDOWS  95 
Cyberlink  Mouse  Driver  -  the  C.A.T.  (Cyberlink  Actuated  Tracker) 

2)  Input  devices  (sample  of  common  manual  mouse  devices): 

a)  Dexxa  two  button  Mouse, 

b)  Logitech  three-button  PS2  Mouse, 

c)  Toshiba  Tecra  ‘Touchstick’  Mouse, 

d)  AST  Ascentia  A  ‘Touchpad’  Mouse 

3)  Performance  Test  Program:  “Fitts  Law”  Target  Acqusition  Task 

Experimental  Design  and  Procedures  -  The  experiment  was  designed  to  compare  the 
performance  of  common  manual  mouse  devices  with  the  C.A.T.  on  a  target  acquisition  and  click- 
confirmation  task.  Independent  variables  were  computer  input  device  (one  or  more  of  four 
possible  manual  mouse  devices,  randomly  balanced;  or  the  C.A.T.),  target  size  (16  x  16,  32  x  32, 
or  48  x  48  pixels)  target  distance  (four  possible  on  the  vertical  and  five  on  the  horizontal  and 
diagonal  axes),  target  direction  angle  (0,  45,  90,  135, 180,  225, 270,  &  315  degrees  as  measured 
counter  clockwise  from  3  o’clock)  and  trial  block.  Subjects  were  trained  as  near  to  asymptotic 
performance  as  practical,  given  study  time  constraints,  on  each  input  device  used  in  the 
experiment  prior  to  running  their  experimental  trials. 

Each  session  consisted  of  six  ‘games’  in  a  random  sequence:  one  game  with  each  of  the  three 
target  sizes  and  two  input  devices  (the  C.A.T.  plus  one  of  the  manual  mouse  devices). 

Note:  some  subjects  performed  additional  three  game  sessions  of  the  task  with  another  of  the 
manual-mouse  devices  to  achieve  balanced  representation  of  each  of  these  devices  in  the  sample. 

“Fitts  Law”  Target  Acquisition  Task  -  Each  game  consisted  of  38  trials,  one  for  each  possible 
target  location.  Each  target  trial  required  the  subject  to  move  the  cursor  from  the  center  of  the 
screen  to  the  target  location  and  issue  a  left  mouse  button  click  command  while  within  the 
boundaries  of  the  target  to  confirm  acquisition  (Figure  8).  The  cursor  then  automatically  returned 
to  the  green  square  in  the  center  of  the  screen  for  two  seconds  after  which  the  next  target 
appeared. 
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Figure  8.  All  Possible  Target  Locations  in  the  Fitts  Law  Target  Acquisition  Task 

The  first  target  appearance  was  initiated  by  clicking  within  the  center  green  box.  The  38  targets 
appeared  in  a  random  order.  The  arrow  cursor  was  replaced  with  crosshairs  while  within  the 
target’s  boundaries.  A  programmed  intermission  was  provided  as  a  rest  period  after  the 
appearance  of  the  19th  target.  Subjects  were  given  no  special  coaching  with  regard  to  specific 
cursor  control  techniques. 

The  following  data  was  automatically  stored  as  a  text  file  and  imported  into  MS  EXCEL  for 
graphic  display  and  analysis:  1)  trial  number  (1  -38),  2)  target  acquisition  time  (in  seconds), 

3)  Fitts  Index  of  Difficulty  or  log2  [twice  the  target  distance/target  width],  4)  target  distance  (in 
pixels),  5)  target  size  (16,  32,  or  48  pixels),  6)  target  angle  (in  degrees  counterclockwise  from 
3  o’clock)  7)  click  commands  off  target,  8)  time  (in  seconds)  to  first  target  contact  with  cursor, 
and  9)  target  number  (1  -38). 

5.2  Results 

Paul  M.  Fitts  derived  the  Fitts  Law  (Fitts,  1954)  from  Shannon’s  Theorem  17  of  information 
transmission  (Shannon  and  Weaver,  1949)  to  predict  human  movement  times.  Movement  time 
(in  this  case,  target  acquisition  time)  is  expected  to  increase  proportionally  with  Index  of 
Difficulty  (ID)  defined  as 

ID  =  log2  (2D/W) 

where  D  is  distance  to  target  and  W  is  target  width. 
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This  model  has  been  applied  successfully  in  several  recent  comparison  performance  studies  of 
manual  and  alternative  cursor  control  devices  (i.e.,  Borah,  1995  and  Epps,  1986). 

The  target  acquisition  time  versus  Fitts  ID  scatter  plot  of  all  trials  using  the  C.A.T.  input  device 
(Figure  9)  yields  a  relatively  steep  regression  slope.  This  indicates  a  strong  relationship  between 
acquisition  time  and  Fitts  ID  in  good  accord  with  Fitts  Law .  The  targets  with  the  highest  Fitts 
ID  values  took  more  than  twice  as  long,  on  average,  to  acquire  as  the  targets  with  the  lowest  Fitts 
ID  values. 

The  scatter  plot  of  the  data  from  the  balanced  sample  of  manual  mouse  devices  shows  a  more 
shallow  regression  slope,  yet  one  consistent  with  both  the  findings  of  other  manual  mouse  studies 
(Borah,  1995  and  Epps,  1986)  and  the  predictions  of  Fitts  Law. 

Figure  10  shows  the  separate  effects  of  target  size,  distance,  and  angle  on  acquisition  time.  With 
both  the  C.A.T.  and  the  manual  mouse  devices,  the  smallest  targets  (16  x  16  pixels)  are 
associated  with  a  mean  acquisition  time  disproportionately  greater  than  those  obtained  with  the 
two  larger  target  sizes. 

In  the  case  of  the  manual  mouse  devices,  this  disparity  does  not  present  a  practical  problem  for 
the  user  because  average  acquisition  time  for  the  16x16  pixel  target  is  less  than  two  seconds.  On 
the  other  hand,  16x16  pixel  mean  acquisition  time  with  the  C.A.T.  exceeds  eight  seconds.  This 
suggests  that  the  WINDOWS  desktop  minimum  button  sizes  may  need  to  be  expanded  (easily 
accomplished  in  the  WINDOWS  “control  panel”)  for  the  C.A.T.  to  be  an  effective  input  device 
in  operational  settings. 

The  dependence  of  acquisition  time  on  target  distance  (Figure  10)  is  slightly  more  pronounced 
with  the  C.A.T.  than  with  the  manual  mouse,  but  the  similarity  of  these  results  suggests  that  the 
C.A.T.  may  handle  distances  in  a  manner  comparable  to  the  manual  mouse.  Acquisition  time 
appears  to  be  slightly  more  influenced  by  target  size  (especially  the  16  x  16  pixel  target)  than  by 
target  distance  for  the  manual  mouse,  and  perhaps  also  for  the  C.A.T. 

Acquisition  time  is  shown,  in  Figure  10  (bottom  graph),  to  be  less  dependent  on  target  angle  than 
on  the  other  factors  except  in  the  case  of  the  270°  target  trials  with  the  C.A.T.  These  targets  were 
directly  below  the  cursor  starting  position  and  required  only  relaxation  of  the  facial  muscles 
(causing  the  cursor  to  drop)  and  very  slight  eye  movements  (to  maintain  lateral  position)  for  the 
target  to  be  intercepted.  More  active  x-y  control  inputs  were  necessary  for  the  other  C.A.T.  target 
angles. 

The  average  interval  between  first  cursor  contact  with  the  target  boundary  and  confirmation  of 
target  acquisition  (via  EMG  click  with  the  C.A.T.  and  left  mouse  button  with  the  manual 
devices)  was  approximately  20%  of  the  total  target  acquisition  time  in  both  the  C.A.T.  and  the 
manual  mouse  trials.  This  worked  out  to  be  approximately  1.3  seconds  for  the  C.A.T.  and 
.3  seconds  for  the  manual  mouse.  The  delay  reflects  a  combination  of  target  overshooting  (less  a 
factor  with  the  manual  mouse)  and  reaction  time  (less  a  factor  with  the  C.A.T.). 
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Figure  9.  Target  Acquisition  Time  Versus  Fitts  Index  of  Difficulty  for  Balanced  Sample  of  Common  Manual 
Mouse  Devices  and  the  Cyberlink  Actuated  Tracker  (C.A.T.) 
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Figure  10.  Factors  Affecting  Target  Acquisition  Time  for  Balanced  Sample  of  Common  Manual 
Mouse  Devices  and  Cyberlink  Actuated  Tracker  (C.A.T.) 


5.3  Discussion 


On  average,  the  C.A.T.  was  approximately  4  times  slower  in  the  target  acquisition  task  than  the 
manual  mouse.  This  difference  is  affected  to  some  extent  by  the  substantial  manual  mouse 
training  advantage  enjoyed  by  the  subjects.  All  could  claim  a  minimum  of  eight  years  PC 
computer  mouse  experience  and  no  more  than  a  few  weeks  experience  with  the  C.A.T.  controller 
used  in  this  experiment. 

Based  on  their  training  logs,  it  is  likely  that  the  naive  users  hadn’t  achieved  full  asymptotic 
performance  with  the  C.A.T.  and  might  have  reduced  their  acquisition  times  by  an  additional  10 
or  20%  had  a  few  more  weeks  been  available  for  training. 

Several  subjects  reported  that,  at  first,  the  use  of  the  Cyberlink  interface  in  the  task  required  more 
conscious  effort  than  the  manual  mouse.  Eventually,  techniques  and  strategies,  such  as  when  to 
initiate  C.A.T.  cursor  speed  changes,  began  to  feel  more  automatic  and  subconscious. 

Further  studies  with  longer  training  periods  may  shed  additional  light  on  the  practical  limits  of 
Cyberlink  interface  speed,  accuracy,  and  efficiency  in  time-intensive  cursor  positioning  plus 
clicking  tasks.  Nevertheless,  the  results  of  this  test  would  seem  to  indicate  a  very  real  potential 
for  a  viable  alternative  computer  input  device. 


5.4  Conclusion 

The  findings  of  the  Phase  II  effort  have  direct  application  to  human-machine  interaction.  In 
addition  to  bringing  the  B.A.T.  technology  from  proof-of-concept  to  preproduction  prototype,  the 
effort  provided  a  new  understanding  of  how  humans  interface  with  machines  specifically  through 
new  control  channels  such  as  those  obtained  with  the  Cyberlink  system. 

In  the  military  sector  the  Cyberlink  interface  could  find  its  way  into  an  aircraft  cockpit  as  an 
alternative  control  input  for  non-critical  flight  systems  such  as  avionics  or  radar  displays  to  off¬ 
load  some  discrete  manual  control  functions.  At  this  time,  Cyberlink  x-y  control  does  not  appear 
to  be  rapid  enough  for  the  primary  flight  control  environment. 

On  the  other  hand,  the  Cyberlink  technology  may  provide  a  means  to  achieve  limited  aircraft 
control,  not  as  a  primary  system  but  as  an  alternative  or  backup  system.  In  the  event  a  pilot  was 
severely  disabled  due  to  injury  or  catastrophic  failure,  a  system  developed  from  the  Cyberlink 
interface  might  provide  the  means  by  which  the  pilot  could  continue  to  control  the  aircraft  or 
could  liberate  himself/herself  from  the  aircraft. 

Applications  are  ripe  for  development  of  the  Cyberlink  C.A.T.  as  a  hands-free  mobile  computer 
interface  in  both  the  military  and  civilian  sectors.  It  is  particularly  well  suited  for  this  application 
because  its  multiple  discrete  command  functionality  may  augment  or  replace  voice  command 
inputs  in  noisy  work  environments. 
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In  military  and  civilian  sectors,  the  Cyberlink  interface  has  great  potential  for  assisting 
individuals  who  are  severely  disabled.  It  can  provide  hands-free  assistive  technology  interfaces, 
environmental  control,  computer  access,  and  recreation  enabling  an  increased  sense  of 
independence  and  quality  of  life  for  individuals  with  limited  motor  control.  It  promises  to  be 
particularly  well  adapted  to  special  needs  word  processing,  hands-free  WEB  browsing  and  other 
internet  applications,  paving  the  way  for  revolutionary  developments  in  facilitated 
communications. 

In  the  field  of  education,  the  Cyberlink  interface  provides  easy  and  rapid  access  to  a  wide  range 
of  biological  signals  for  self  discovery,  research,  relaxation/stress  reduction,  and  group  multi- 
media  biofeedback  events. 

In  the  video  game  market,  leading  software  developers  and  publishers  perceive  the  B.A.T. 
technology  as  a  possible  means  to  extend  the  lifecycle  of  their  current  products  by  adding 
"thought  control  powers"  and  to  increase  the  immersive/interactive  characteristics  of  existing 
games.  Others  see  the  opportunity  to  expand  their  lines  into  a  leading  edge  technology  for 
personal,  arcade,  or  other  on-site  entertainment  centers. 
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